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Abstract 

The  validation  between  a  research  and  developmental  satellite  and  its  ground  system 
is  critical  to  ensuring  the  success  on-orbit.  However,  the  exact  process  for  completing 
validation  is  not  documented,  frequently  underfunded,  and  accomplished  ad  hoc.  This 
leads  to  debate  regarding  maintenance  of  budget  and  schedule,  while  ensuring  on-orbit 
success. 

This  thesis  examines  readiness  and  on-orbit  activities  within  the  U.S.  Air  Eorce  Space 
Development  and  Test  Wing’s  Research  Development  Test  and  Evaluation  Support 
Complex.  Combining  historical  data  with  the  consultation  of  subject  matter  experts,  a 
validation  process  was  defined.  Risks  associated  with  this  process  were  then  analyzed 
using  the  Strategy  Based  Risk  Model,  and  were  evaluated  based  on  the  probability  of 
occurrence  and  severity  of  impact.  The  validation  process  and  associated  costs  were 
validated  using  the  Delphi  Method.  Next,  we  transformed  the  results  into  a  simulation 
that  generates  distributions  of  possible  costs  and  risk  outcomes.  Einally  we  applied  the 
simulation  to  a  program,  and  distributed  it  to  program  managers  for  feedback.  The 
simulation  will  be  distributed  to  program  offices  to  support  tailoring  a  validation  plan 
relative  to  their  budget.  The  simulation  will  give  decision  makers  greater  fidelity  into  the 
expected  risks  and  costs  associated  with  the  selected  validation  process. 
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EXTENDING  THE  STRATEGY  BASED  RISK  MODEE  USING  THE  DELPHI 
METHOD:  AN  APPLICATION  TO  THE  VALIDATION  PROCESS  EOR  RESEARCH 
AND  DEVELOPMENTAL  (R&D)  SATELLITES 


I.  Introduction 


1.1  Background 

Prior  to  launching  a  satellite,  the  satellite  and  ground  system  must  undergo  a  series  of 
validation  tests  to  ensure  they  are  capable  of  communicating  with  one  another.  The 
validation  of  the  compatibility  between  a  satellite  and  its  ground  system  is  comprised  of 
three  main  events:  a  compatibility  test  between  the  satellite,  the  ground  system,  and  the 
Command  Control  and  Communication  (C3)  node  (e.g.  Air  Eorce  Satellite  Control 
Network  (AESCN),  Tracking  and  Data  Relay  Satellite  System  (TDRSS),  or  other), 
database  (command  and  telemetry)  validation,  and  data  flow  testing.  Although  thorough 
verification,  validation,  and  testing  processes  have  been  laid  out  between  large  budget 
satellite  programs  and  their  ground  systems,  no  such  process  has  been  defined  for 
Research  &  Developmental  (R&D)  satellite  missions.  Therefore,  organizations  that 
specialize  in  flying  unique  R&D  satellites  are  presented  with  specific  challenges  because 
there  is  no  standardized  approach  for  validating  the  ground  systems.  Since  there  is  no 
standardization  of  the  system  validation  process,  every  satellite  program  varies  or 
modifies  its  own  validation  plan.  Additionally,  the  limited  budgets  of  most  R&D  satellite 
program  offices  contribute  to  the  widely  varying  validation  plans. 
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Furthermore,  each  stakeholder  has  a  different  methodology  for  conducting  validation 
testing.  An  example  of  this  difference  is  apparent  between  the  Space  Development  and 
Test  Wing  (SDTW)  and  the  Air  Force  Research  Laboratory  (AFRL).  SDTW  specializes 
in  space  technology  demonstrations;  this  includes  flying,  launching  and  funding 
experimental  satellites  missions.  A  large  quantity  of  these  experimental  satellite  missions 
are  developed  and  funded  in  part  by  AFRL,  making  them  one  of  SDTW’s  largest 
customers.  SDTW  and  AFRL  frequently  disagree  in  regard  to  their  stance  on  Launch 
Based  Compatibility  Tests  (LBCT).  SDTW,  along  with  their  parent  organization.  Space 
and  Missile  Systems  Center  (SMC),  under  the  control  of  Air  Force  Space  Command 
(AFSPC)  requires  an  LBCT.  AFRL,  under  Air  Force  Materiel  Command  (AFMC), 
frequently  deems  an  LBCT  as  unnecessary.  This  philosophical  difference  stems  from 
AFRL  working  primarily  in  R&D,  while  most  of  AFSPC  works  with  operational 
satellites.  Agencies  are  more  likely  to  take  risks  with  R&D  satellites  than  the  operational 
satellites  upon  which  our  military  and  country  depend.  Although  the  philosophical 
differences  are  well  understood,  it  causes  conflict  within  the  organizations  because 
AFMC  has  satellite  control  authority  (SCA),  yet  AFSPC  operates  the  satellites. 

Conflicts  like  the  one  presented  above  stem  from  a  difference  in  stakeholder  priorities. 
Each  satellite  mission  has  a  different  set  of  stakeholders.  The  stakeholders  have  different 
methods  for  validating  compatibility  between  the  satellite  and  the  ground  system.  These 
differences  arise  from  adversity  to  risk,  a  direct  result  of  their  different  and  limited 
budgets.  Nobody  wants  to  put  a  satellite  into  orbit  that  cannot  communicate  with  the 
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ground  system,  but  the  offiee  paying  the  validation  bill  will  question  the  neeessity  of  the 


full  validation  proeess.  There  are  numerous  ways  to  reduee  eosts  during  the  validation 
proeess,  namely  the  reduetion  of  validation  events.  However,  eaeh  skipped  event  adds 
risk  to  the  program.  Diseeming  the  potential  program  risks  helps  stakeholders  determine 
the  optimal  steps  in  the  validation  proeess  for  the  program  based  on  their  budget. 
Validation  oecurs  late  in  the  lifeeyele  of  a  program,  thus  most  of  the  budget  reserve  has 
been  eonsumed.  As  a  result,  this  is  the  least  opportune  time  for  a  program  to  eneounter  a 
problem. 

1.2  Scope  and  Purpose 

The  purpose  of  this  thesis  was  to  develop  a  simulation  that  generates  distributions  of 
possible  risk  and  eost  impaets.  This  simulation  will  aide  R&D  satellite  program  offiees 
in  identifying  the  eritieal  steps  in  the  proeess  of  the  validating  the  eompatibility  between 
a  satellite  and  its  ground  system.  This  model  will  help  determine  the  neeessary  validation 
steps,  while  also  determining  possible  steps  to  eliminate  to  balance  cost  and  risk.  For  this 
thesis,  validation  refers  to  those  steps  that  ensure  the  correct  system  was  built. 

Verification  ensures  the  system  was  built  correctly.  Verification  is  not  within  the  scope 
of  this  thesis.  Validation  determines  the  correctness  and  completeness  of  the  end 
product,  and  ensures  the  system  satisfies  the  needs  of  the  stakeholders  [Bahill  & 
Henderson,  2004] .  R&D  satellites  frequently  remove  and/or  modify  steps  during  the 
validation  of  the  satellite  and  its  ground  system  to  meet  budget.  Stakeholders  can  use  this 
simulation  to  support  discussions  on  balancing  the  validation  effort  with  cost  and  risk. 
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1.3  Research  Questions 


This  thesis  answered  the  following  questions  in  order  to  develop  a  simulation  that 
generates  distributions  of  possible  costs  and  risk  outcomes: 

•  What  are  the  steps  that  need  to  be  carried  out  as  part  of  the  process  for 
validating  the  compatibility  between  the  satellite  and  its  ground 
system? 

•  What  are  the  costs  to  complete  each  step  of  the  validation  process? 

•  What  are  the  risks  associated  with  not  completing  each  step  of  the 
validation  process? 

•  What  is  the  desired  balance  between  cost  and  risk  for  a  given 
validation  strategy? 

These  outcomes  were  analyzed  in  relation  to  the  impact  events  associated 
with  the  realization  of  risks. 

1.4  Methodology 

Our  thesis  methodology  is  based  on  Avner  Engel  and  Miryam  Barad’s  paper,  A 
Methodology  for  Modeling  Verification  Validation  and  Testing  (VVT)  Risks  and  Costs. 

In  this  paper  Engle  and  Barad  examine  the  risks  and  costs  associated  with  VVT  for  the 
Israeli  aircraft  industry  [Engel  &  Barad,  2003].  We  have  adapted  their  methodology  to  fit 
our  model  for  the  validation  of  the  compatibility  between  R&D  satellite  and  their  ground 
systems. 
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The  methodology  was  divided  into  four  distinct  steps.  During  the  first  two  steps  we 
relied  on  a  panel  of  experts  to  provide  data.  The  first  step  was  broken  into  two  parts:  (1) 
develop  a  model  of  all  of  the  steps  associated  with  validating  the  compatibility  between 
an  R&D  satellite  and  its  ground  system,  and  (2)  assign  appropriate  costs  to  each  activity. 
In  the  second  step  we  identified  the  program  risks  that  are  mitigated  by  executing  the 
steps  in  the  validation  process  and  the  costs  associated  with  the  impacts  of  those  risks. 
The  third  step  in  our  methodology  used  the  data  collected  in  the  previous  steps  to  create  a 
simulation  that  generates  distributions  of  possible  costs  and  risk  outcomes.  This 
simulation  aides  the  program  office  when  building  a  validation  plan  and  justifying  the 
validation  plan  to  leadership.  The  program  office  is  able  to  review  all  possible  risks 
concerning  the  ground  system  and  satellite  validation  plan,  along  with  the  severity  and 
probability  of  the  risks.  Next,  they  can  review  the  validation  steps  that  mitigate  the  risks 
and  their  costs.  Finally,  they  review  the  generated  distributions  to  build  their  validation 
plan  and  allow  for  risk  realization  based  on  their  budget  and  the  risks  they  wish  to 
mitigate.  The  fourth  and  final  step  in  our  methodology  was  to  demonstrate  the  usability 
of  our  simulation  by  applying  it  to  an  on-going  R&D  program.  Additionally,  we  had  two 
program  managers  apply  the  technique  and  simulation  to  their  programs.  This  was  done 
to  validate  that  the  technique  and  simulation  were  easily  understood  and  could  be  applied 
by  program  managers  that  did  not  have  insight  into  our  thesis. 
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1.5  Assumptions  and  Limitations 

Engel  and  Barad  made  a  number  of  assumptions  when  conducting  their  research  on 
the  Israeli  aircraft  industry’s  VVT  efforts.  As  our  thesis  is  based  on  the  works  of  Engel 
and  Barad,  we  made  many  of  the  same  assumptions.  Eirst,  we  assumed  that  the 
Canonical  Verification,  Validation  and  Test  Model  (CVM)  is  a  sequential  process  that 
assumes  linear  progression  of  steps.  Second,  we  assumed  that  all  of  the  validation  steps 
take  place  within  the  same  phase  of  the  mission  lifecycle  -  the  readiness  phase.  Third, 
we  assumed  that  the  risk  impact  costs  and  probabilities  of  all  risk  sources  are 
independent.  Einally  we  assumed  that  each  validation  step  is  either  performed  or  not 
performed,  and  that  a  step  may  not  be  partially  completed  [Engel  &  Barad,  2003]. 

Additionally,  we  made  a  number  of  other  assumptions  in  our  thesis  that  Engel  and 
Barad  did  not.  The  first  assumption  was  that  the  same  basic  risk  areas  apply  to  all 
satellites  and  ground  systems.  As  all  R&D  satellites  investigated  during  this  thesis  use 
the  same  mechanisms  for  communication,  their  risks  areas  will  be  the  same.  Next  we 
assumed  that  the  impact  of  each  risk  is  represented  with  a  dollar  value.  Sometimes  the 
impact  of  a  risk  being  realized  is  a  schedule  slip.  However,  for  the  purpose  of  this  thesis 
we  only  tracked  budgetary  concerns;  therefore,  a  schedule  slip  was  correlated  to  the 
monetary  cost  associated  with  it.  Einally  it  was  determined  that  each  risk  is  mitigated  by 
at  least  one  step  in  the  validation  process. 

We  also  encountered  limitations  to  our  research.  Eirst,  our  data  only  examined 
validation  (end  to  end  testing)  and  not  inspection  or  system  level  testing.  Inspection  and 
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system  level  testing  oeeur  prior  to  validation  testing  in  the  systems  engineering  proeess 
and  thus  was  not  eonsidered  in  this  thesis.  Next,  all  satellite  systems  we  reviewed  during 
the  eourse  of  this  thesis  eommunieated  via  the  AFSCN.  However,  satellites  utilizing 
other  eommunieations  meehanisms  would  follow  similar  validation  steps  to  ensure 
eompatibility  between  the  satellite  and  its  ground  system.  Additionally,  beeause  CVM  is 
a  sequential  proeess  that  assumes  linear  progression  of  steps,  CVM  does  not  aeeount  for 
re-exeeution  of  steps  due  to  failure.  Another  limitation  in  our  thesis  was  due  to  the 
availability  of  information.  Moreover,  only  four  programs  were  used  to  derive  the  eost 
data  for  the  validation  steps.  The  teehnique  developed  in  our  thesis  examined  how  the 
validation  steps  can  reduce  the  probability  of  a  risk  being  realized,  the  steps  in  the 
validation  process  do  not  mitigate  the  severity  of  the  realization  of  risks.  Finally,  risks 
can  be  defined  as  threats  and  opportunities;  however  this  thesis  only  examined  threats. 
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II.  Background 


2.1  Validation  in  the  System’s  Engineering  Process 

According  to  Dennis  M.  Buede,  author  of  The  Engineering  Design  of  Systems:  Models 
and  Methods,  Systems  Engineering  is  defined  as:  “an  interdisciplinary  approach 
encompassing  the  entire  set  of  scientific,  technical,  and  managerial  efforts  needed  to 
evolve,  verify,  deploy  (or  field),  and  support  an  integrated  and  life-cycle  balanced  set  of 
system  solutions  that  satisfy  customer  needs”  [Buede,  2000].  The  Systems  Engineering 
Process  is  an  important  part  of  any  development  effort,  but  it  is  especially  important  for 
R&D  satellite  program  offices  because  of  their  limited  budgets.  One  of  the  most 
accepted  models  of  the  systems  engineering  process  is  the  Vee  Model.  Eigure  1  shows  a 
typical  system  development  lifecycle  as  a  “Vee”  with  the  emphasis  of  the  model  from  a 
systems  engineering  perspective.  The  left,  or  decomposition,  side  of  the  Vee  illustrates 
the  phases  at  the  beginning  of  a  typical  system  lifecycle,  and  focuses  on  requirements 
definition  and  development  of  the  system  specification.  The  bottom  of  the  Vee  develops 
system  specifications  into  a  build-to  design  and  the  resulting  products  product.  The  right 
side  of  the  Vee  depicts  the  final  steps  in  system  development. 
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Figure  1:  Systems  Engineering  Vee  [Buede,  2000] 


These  steps  focus  on  ensuring  the  system  meets  the  requirements  and  user  needs.  This 
process  is  Verification  Validation  and  Testing  (VVT).  Our  thesis  focused  on  a  particular 
part  of  the  final  step  in  the  Systems  Engineering  Vee:  Demonstrate  and  Validate  System 
to  User.  Specifically  we  defined  our  user  as  the  Research  and  Developmental  (R&D) 
Satellite  Program  Office.  We  demonstrated  and  validated  that  our  System-of-Systems, 
the  satellite  and  ground  system,  are  compatible. 

According  to  Buede,  “validation  of  the  design  problem  demonstrates  as  completely  as 
possible  that  the  design  problem  as  defined  by  a  large  set  of  requirements  is  the  same 
design  problem  as  reflected  in  the  operational  concept  and  in  the  minds  of  the 
stakeholders.”  Validation  illustrates  to  the  stakeholders  that  the  ground  system  design 
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and  integration  meets  the  needs  of  the  satellite  and  that  the  satellite  can  communicate 
with  the  ground  system  [Buede,  2000]. 

2.2  R&D  Satellites 

R&D  satellite  programs  flown  through  the  Space  Development  and  Test  Wing 
(SDTW)  come  from  variety  of  different  sources.  They  can  be  manifested  by  the 
Department  of  Defense  (DoD)  Space  Test  Program  (STP)  through  the  Satellite 
Experiment  and  Review  Board  (SERB).  Programs  that  come  to  STP  are  able  to  request 
help  with  launch,  integration  and/or  operations  costs.  The  programs  requesting  STP 
services  can  be  small  payloads  or  small  satellites  such  as  cubesats,  which  are  10cm 
xlOcm  satellites.  STP  also  supports  large  complex  multi-satellite  systems.  R&D  satellite 
programs  across  DoD  use  the  SERB  to  help  find  flights  to  space  due  to  their  own  funding 
shortages  and  DoD  STPs  ability  to  engineer  partnerships  for  successful  spaceflight. 

When  DoD  STP  services  (funds)  are  requested  for  operations,  the  satellites  are 
operated  within  SDTW.  Similarly,  when  AE  R&D  organizations,  such  as  the  Air  Eorce 
Research  Eaboratory  (AERE),  look  to  outside  agencies  to  fly  their  satellites,  they  also 
come  to  SDTW  on  a  cost  reimbursable  basis. 

R&D  satellites  come  in  a  variety  of  shapes,  sizes,  and  budgets.  A  cubesat  is  typically 
about  $300K,  and  the  most  expensive  R&D  satellite  mission  to  fly  out  of  SDTW  in  recent 
years  was  over  $400M. 

As  R&D  satellite  size  can  vary  depending  on  mission,  so  can  the  complexity.  Some 
satellites  conduct  simple  operations  of  scientific  payloads  and  have  basic  operational 
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concepts.  Others  are  much  more  complicated  and  thus,  the  operating  system  and  ground 
system  are  much  more  complex.  The  complexity  of  the  mission  will  affect  how  thorough 
some  steps  in  the  validation  process  are  and  how  long  they  will  take.  As  the  cost  of  each 
validation  step  is  related  to  the  number  of  hours  it  takes  to  complete  the  step,  there  is  a 
direct  correlation  between  the  complexity  of  a  satellite  program  and  the  cost  of  the 
validation  effort. 

2.2.1  R&D  Satellite  Validation 

There  are  many  similarities  between  Operational  Satellite  VVT  and  Experimental 
Satellite  VVT.  Both  operational  and  experimental  satellites  undergo  rigorous  testing  to 
ensure  the  mission’s  success.  They  both  complete  basic  testing,  to  include: 
environmental,  thermal,  and  basic  compatibility  testing.  However,  the  testing  does  not 
always  have  the  same  focus  area.  Operational  satellites  are  part  of  a  constellation 
whereas  most  experimental  satellites  are  one-of-a-kind.  Previous  satellites  in  the 
constellation  have  already  validated  the  compatibility  between  operational  satellites  and 
their  ground  system.  Therefore  the  focus  of  the  operational  satellite  testing  is 
sustainability  and  ensuring  that  planned  redundancy  will  work  for  the  mission. 
Sustainability  is  comprised  of  reliability,  maintainability  and  availability  (RMA)  of  both 
the  satellite  and  the  ground  system.  This  validation  activity  ensures  the  new  satellite  is 
the  same  as  the  previous  satellites  in  the  constellation.  Testing  for  RMA  of  experimental 
satellites  is  not  possible  because  this  type  of  testing  requires  previous  data  for 
comparison.  Operational  satellites  do  not  have  Week  in  the  Life  (WITL)  or  Day  in  the 
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Life  (DITL)  tests.  Additionally,  the  command  and  telemetry  databases  have  already  been 
validated.  Operational  satellites  also  have  significantly  fewer  training  events  since  the 
operators  know  how  to  fly  the  satellite.  Most  experimental  satellites  have  at  least  six 
exercises  and  rehearsals,  while  operational  satellites  will  only  have  two  training  events. 
These  events  focus  on  the  launch  and  initialization  sequences.  Additional  information  on 
these  steps  is  included  in  the  methodology  and  analysis  &  results  sections  [Trautwein, 
2009]. 

As  briefly  mentioned  in  Section  1.1,  Air  Force  Space  Command  (AFSPC)  requires  a 
rigorous  set  of  validation  tests  because  the  majority  of  their  programs  have  multi -billion 
dollar  budgets.  In  the  R&D  arena,  there  are  a  number  of  different  stakeholders  and  the 
stakeholders  involved  all  come  prepared  to  fight  for  the  validation  plan  their  leadership 
favors.  The  first  set  of  stakeholder  is  the  ground  system  developer/operator,  in  this  case 
SDTW.  Next  is  the  experiment  owner,  also  known  as  the  satellite  program  office.  The 
satellite  program  office  typically  has  satellite  control  authority  and  wants  to  see  a 
successful  mission.  The  satellite  program  office  is  typically  most  concerned  with  the 
validation  budget.  The  final  stakeholder  is  the  satellite  manufacturer,  who  needs  a 
successful  mission  to  generate  future  revenue,  but  has  little  input  into  the  overall 
validation  plan.  The  overall  decision  on  the  satellite/ground  system  validation  plan  is 
made  between  the  ground  system  developer/operator  and  the  satellite  program  office. 

The  cost  of  large  systems  VVT  is  approximately  40%  of  the  total  life  cycle  cost  of  the 
system  [Engel  &  Barad,  2003].  However,  R&D  satellite  program  offices  do  not  have 


12 


large  budgets  and  therefore  cannot  spend  40%  of  their  budget  on  VVT.  Engel  and 
Barad’s  paper  on  methodology  for  risk  and  cost  monitoring  for  VVT  proposes  a  novel 
approach  for  modeling  VVT  strategies  as  decision  problems.  This  paper  only  addresses 
the  VVT  issue  in  regards  to  large  aircraft  programs  [Engel  &  Barad,  2003].  In  this  thesis, 
we  took  their  study  and  focused  on  the  validation  of  the  compatibility  between  R&D 
satellites  and  their  ground  systems.  Within  validation  some  steps  are  required  and  the 
costs  are  the  same  regardless  the  size  of  the  program.  Other  steps  can  be  modified  to  fit 
the  size  of  the  program  and  the  level  of  risk  the  program  is  willing  to  accept  [Engel  & 
Barad,  2003]. 

2.3  Engel  and  Barad’s  Methodology  for  Modeling  WT  Risks  and  Costs 

Throughout  Engel  and  Barad’s  research  they  found  that  most  modeling 
methodologies  have  two  significant  weaknesses.  These  weaknesses  are  that  most  models 
are  not  organized,  and  consequently  all  risks  are  not  necessarily  identified.  Additionally, 
they  found  that  cost  estimates  do  not  consider  all  factors,  i.e.  variances  in  cost  estimates 
and  VVT  costs  associated  with  the  life  cycle  of  the  system.  In  order  to  counter  this,  they 
developed  a  process  that  accounted  for  these  weaknesses,  yielding  an  advanced  model. 
This  process  includes:  defining  a  canonical  model,  modeling  VVT  strategy  as  a  decision 
problem,  and  developing  a  strategy-based  VVT  process  for  risk,  cost,  and  performance 
duration. 
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2.3.1  Defining  a  Canonical  VVT  Model 

The  canonical  data  model  is  the  standard  organizational  view  on  a  particular  subject, 
mapping  back  to  each  application  view  on  the  same  subject.  The  standard  organizational 
view  is  built  traditionally  using  simple  yet  useful  structures  [Hoberman,  2008].  The 
Canonical  VVT  Model  (CVM)  assumes  each  activity  associated  with  the  validation  effort 
occurs  sequentially.  Figure  2  depicts  the  life  cycles  phases,  activities,  costs,  and  timeline 
elements  of  the  CVM.  This  model  should  only  be  used  to  evaluate  partial  sets  of 
activities  in  relation  to  the  full  set.  This  model  was  appropriate  for  our  research  because 
we  researched  a  specific  part  of  the  systems  engineering  process  as  it  applies  to  small 
satellite  missions. 
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Figure  2:  Canonical  VVT  Model  [Engel  &  Barad,  2003] 


2.3.2  VVT  Strategy  as  a  Decision  Problem 

Engel  and  Barad  recognized  that  executing  the  all  inclusive  CVM  is  not  practical  due 
to  budgetary  constraints.  Therefore  to  account  for  this  reality,  the  VVT  strategy  must  be 
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considered  a  decision  problem  [Engel  &  Barad,  2003].  In  order  to  do  this,  Engel  and 
Barad  introduced  some  basic  concepts.  The  first  was  to  define  VVT  Strategy.  “A  VYT 
strategy  is  a  policy  for  a  given  system  life  cycle,  under  which  a  subset  of  steps  are  fully 
performed  another  subset  of  steps  is  partially  performed  and  the  remaining  activities  are 
not  performed  at  all”  [Engel  &  Barad,  2003].  The  other  concept  Engel  and  Barad 
introduced  was  the  decision  variable.  As  stated  above,  every  step  in  the  VVT  model  can 
be  fully  performed,  partially  performed  or  not  performed  at  all.  The  decision  variable  is 
the  performance  level  of  any  step  in  the  VVT  process.  The  value  of  the  decision  variable 
is  always  0,  1,  or  somewhere  in  between.  A  value  of  0  indicates  that  the  VVT  step  was 
not  performed.  Similarly  a  value  of  1  indicates  that  the  VVT  step  was  fully  performed. 
Any  value  between  0  and  1  indicates  partial  performance  of  a  validation  step.  As  stated 
in  the  assumptions  in  Chapter  One,  for  the  scope  of  this  thesis  we  will  assume  that  the 
decision  variable  is  either  0  or  1. 

2.3.3  Developing  a  Strategy  Based  Risk  Model 

Throughout  their  research  Engel  and  Barad  used  the  Strategy  Based  Risk  Model 
(SRM)  to  create  their  risk  mock-up.  Before  we  examine  SRM,  it  is  important  to 
understand  the  definition  of  risk.  “A  risk  is  defined  as:  any  uncertainty  that,  if  it  occurs, 
would  have  a  positive  or  negative  effect  on  achievement  of  one  or  more  objectives.” 

Risk  includes  both  threats  and  opportunities  [Hillson  &  Simon,  2007].  “Risk 
management  is  defined  as:  the  structured  process  of  making  appropriate  decisions  and 
implementing  actions  in  response  to  known  risk  events  and  overall  project  risk”  [Hillson 
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&  Simon,  2007].  Risk  is  then  considered  a  cost  driver  because  managing  risk  creates  cost 


and  any  risk  that  comes  to  fruition  will  either  impact  the  schedule  and/or  the  problem  will 
need  to  be  resolved  thus  having  cost  implications.  Engel  and  Barad  define  SRM  as  a 
model  for  discerning  risk,  probability  of  impact  and  cost  of  impact  for  a  given  VVT 
strategy.  The  SRM  concept  comprises  “risk  identification  attributes”  and  “risk 
variables.”  The  risk  attributes  include,  the  risk  source  and  risk  destination.  The  risk 
source  is  a  qualitative  description  of  the  risk.  The  risk  destination  is  a  step  in  the 
validation  process  that  is  designed  to  address  the  risk.  The  two  risk  variables  are  the 
probability  that  the  risk  will  impact  the  mission,  and  the  severity  of  that  impact  [Engel  & 
Barad,  2003]. 

MIE-STD-882C  breaks  the  probability  of  risk  into  five  categories.  Table  1  is 
extracted  from  this  standard  and  provides  guidelines  in  terms  of  the  likelihood  of  the 
occurrence  over  the  lifetime  of  an  item  and  the  likelihood  of  occurrence  per  number  of 
items. 


Table  1:  Probability  of  Risk  Occurrence  [MIE-STD-882C] 


Probability 

description 

Likelihood  of  occurrence 
over  lifetime  of  an  item 

Likelihood  of  occurrence 
per  number  of  items 

Frequent 

Likely  to  occur  frequently 

Widely  experienced 

Probable 

Will  occur  several  times  in  life  of  item 

Will  occur  frequently 

Occasional 

Likely  to  occur  some  time  in  life  of  item 

Will  occur  in  several  items 

Remote 

Unlikely  but  ptossible  to  occur  in  life  of  item 

Unlikely  but  can  reasonably  be 
expected  to  occur 

Improbable 

So  unlikely,  it  can  be  assumed  occurrence 
may  not  be  experienced 

Unlikely  to  occur  but  possible 
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MIL-STD-882C  also  breaks  the  severity  of  risk  into  four  categories.  These  categories 


are:  catastrophic,  critical,  marginal  and  negligible.  Engel  &  Barad  extrapolate  the 
information  out  of  MIL-STD-882C  and  create  Table  2,  which  uses  the  criteria  for  the 
safety  categories  and  adds  parallel  criteria  for  the  additional  categories  [Engel  and  Barad, 
2003]. 


Table  2:  Severity  of  Risk  Effects  [MIE-STD-882C] 


Tmpact 

Impact  levels 

C'ateiEorie.s 

CTatastronhic 

Critical 

IVf  areinal 

Neelieible 

Human 

Salbty 

Death 

Severe  Injury 

Minor  Injury 

Less  than  minor 
iniurv 

Systems 

Safely 

Major  equipment 
loss;  Broad  scale 
maior  damauc 

Small  scale  major 
damage 

Broad  scale 
minor  damage 

Small  scale 
minor  damage 

Kn  V  i  ronmenta  1 
l^amaue 

Severe 

Major 

Minor 

Some  trivial 

Occupational 

Illness 

Severe  &  broad 
scale 

Severe  or  broad 
scale 

Minor  &  small 
scale 

Minor  or  small 
scale 

Financial 

Losses  of  a 
proRram 

Loss  of  program 
funds; 

1 00%  cost  ttrowth 

Funds  reductions; 
50-  1  00%  cost 
growth 

20-50%  cost 
growth 

<  20%  cost 
growth 

Functional 
Performance  of 
a  product 

Design  does  not 
meet  critical 
thresholds 

Severe  design 
deficiencies  but 
thresholds  met 

Minor  design 
flaws,  but 
fixable 

Some  trivial 
“out  of  spec" 
design  elements 

Schedule 
Slippage  of  a 
product 

Slip  reduces  overall 
capabilities 

Slip  has  major 
cost  impacts 

Slip  causes 
internal  turmoil 

Republish 

schedules 

Political  or 
Public  Impact 
of  an  event 

Impact  \Videspread 
(Watergate) 

Significant 
(Tailhook  *■91 ) 

I-lmbaixassment 
($200  hammer) 

Local 

Negative 
impact  due  to 
unidentified 
stakeholders 

Ivfajor  stakeholder 
blocks  program. 
(Israeli  AWACS 
sale  to  China) 

Stakeholder 
requires  product 
modiiications. 
(FAA  disqualilics 
new  aircraft) 

Stakeholder 
requires  minor 
system 
modifications 

Upgrading  sales 
campaign  to 
cover  newly 
recognized 
stakeholders 

Future  losses 
of  potential 
revenues 

Customers 
determined  to 
abandon  product 

Major  market 
share  loss 

Customers 
dissatisfied  with 
product 

A  competitor 
plan  to  develop 
similar  product 

Engel  and  Barad  used  the  SRM  “in  order  to  carry  out  a  qualitative  and  quantitative 
model  of  the  risk  associated  with  a  given  VVT  strategy”  [Engel  &  Barad,  2003]. 


2.4.  Define  Costs  Associated  with  Impacts  of  Risks 

In  Engel  and  Barad’s  paper,  they  defined  two  types  of  quality  costs.  The  term  quality 
cost  is  used  to  define  the  costs  associated  with  risk.  The  first  quality  cost  is  the  cost 
associated  with  the  prevention  of  faults,  these  are  considered  validation  costs.  The 
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second  type  of  quality  costs  are  associated  with  internal  and  external  failures,  the  risk 
impact  costs  [Engel  &  Barad,  2003] .  The  overall  VVT  cost  is  the  cost  of  the  validation 
effort  and  the  risk  impact  costs.  This  is  shown  in  Figure  3. 


Quality  Costs 

1 .  Costs  associated  with 
the  prevention  of  faults  & 
those  associated  with 
the  appraisal  of  product 
quality 


2.  Cost  of  internal  & 

external  failures  - >  Risk  Impact  Costs 

^ Impact_STrategy 


Validation  Costs 

^ll'T_STrat^ 


^  0\-erall_in_Co5tCS)  ^ITTjSrrat^- 


^  Impact _STrategy<^ 


Figure  3:  Quality  Costs  Equation  [Engel  &  Barad,  2003] 


Figure  4  illustrates  that  either  all  of  the  VVT  can  be  modeled,  none  of  it,  or  parts  of  it. 
Neglecting  to  complete  the  entire  life  cycle  portion  will  create  risk  for  the  stakeholders 
[Engel  &  Barad,  2003]. 
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Figure  4:  VVT  Strategy  [Engel  &  Barad,  2003]. 


When  a  particular  validation  activity  is  not  performed,  it  increases  one  or  more  risks. 
Therefore,  a  given  validation  strategy  gives  rise  to  a  collection  of  risks. 
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III.  Methodology 


3.1  Introduction 

This  chapter  presents  the  methodology  that  we  used  to  complete  our  thesis.  In  Engel 
and  Barad’s  study  they  proposed  a  methodology  for  modeling  validation  costs  and  risks. 
They  laid  out  their  methodology  in  four  simple  steps  as  shown  in  the  Figure  5  below 
[Engel  &  Barad,  2003]. 
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Figure  5:  Methodology  for  quantitatively  assessing  system  life  cycle  WT  &  risk  cost 

[Engel  &  Barad,  2003] . 


Our  research  is  based  on  the  research  of  Engel  and  Barad,  so  we  also  broke  our 
methodology  into  four  steps.  The  first  step  was  broken  into  two  parts:  (1)  Develop  a 
model  of  all  of  the  steps  associated  with  validating  the  compatibility  between  a  Research 
and  Developmental  (R&D)  satellite  and  its  ground  system.  (2)  Assign  appropriate  costs 
to  each  activity.  In  the  second  step  we  identified  the  program  risks  that  can  be  mitigated 
by  executing  the  steps  in  the  validation  process.  We  also  defined  costs  associated  with 
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the  impacts  of  the  risks.  Throughout  steps  one  and  two,  we  used  the  Delphi  Method  of 
collecting  and  distilling  knowledge  from  a  group  of  Subject  Matter  Experts  (SMEs)  by 
means  of  a  series  of  surveys.  The  third  step  in  our  methodology  was  to  create  a 
simulation  that  generates  distributions  of  possible  cost  and  risk  outcomes  to  be  analyzed 
in  relation  to  the  information  found  in  steps  one  and  two.  This  technique  will  help 
program  offices  make  informed  decisions  about  how  to  execute  the  validation  process. 
The  fourth  and  final  step  in  our  methodology  was  to  demonstrate  the  accuracy  and 
usability  of  our  simulation. 

3.2  Data  Collection  Using  the  Delphi  Method 

Eieutenant  Commander  Timothy  J.  Gilbribe  performed  his  Air  Eorce  Institute  of 
Technology  (AEIT)  thesis  work  using  the  Delphi  Method.  He  stated  that  when  using  the 
Delphi  Method,  one  can  receive  three  types  of  feedback.  The  experts  can  speculate,  give 
their  opinions,  or  respond  based  on  factual  knowledge  of  the  topic  area  [Gilbribe,  2002]. 

Speculation  or  opinion  can  be  defined  as  beliefs  of  someone  (in  our  case  the  expert) 
based  on  their  experiences.  It  is  important  to  note  that  these  SME  opinions  were 
formulated  primarily  through  career  experiences,  learned  facts,  and  personal  observations 
and  beliefs.  As  a  result,  these  opinions  cannot  be  assumed  to  be  proven  facts,  but  rather  a 
means  of  gathering  a  breadth  of  information  to  help  guide  us  to  the  most  correct 
conclusion.  To  help  discern  and  discount  the  opinions  that  can  best  be  referred  to  as 
inaccurate  “outliers”  we  issued  several  iterations  of  the  survey  to  the  SMEs  [Rayens  and 
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Hahn,  2000].  In-depth  reviews  of  the  responses  allowed  us  to  pinpoint  the  majority 
opinion.  This  helped  guide  our  final  answer. 

3.2.1  The  Delphi  Method 

The  Delphi  Method  is  described  within  Measuring  and  Optimizing  Systems  ’  Quality 
Costs  and  Project  Duration  by  Avner  Engel  and  Shalom  Shachar,  as  a  systematic, 
interactive  interviewing  method  which  relies  on  a  panel  of  independent  experts  [Engel  & 
Shachar,  2005].  The  theory  of  the  Delphi  Method  is  that  a  structured  group  of  experts 
will  come  to  a  more  accurate  “correct”  answer  than  an  unstructured  group.  According  to 
Einstone  and  Turoff,  authors  of  The  Delphi  Method:  Techniques  and  Applications,  the 
Delphi  Method  should  be  considered  a  communication  process.  It  is  particularly  useful 
when  attempting  to  gather  current  or  historical  information  not  accurately  known  or 
available,  evaluating  possible  budget  allocations  and  creating  the  structure  of  a  model 
[Einstone  &  Turoff,  2002].  As  stated  in  chapter  one  of  our  thesis,  there  is  no  process  for 
conducting  validation  of  the  compatibility  of  R&D  satellites  and  their  ground  systems. 
Cost  is  almost  always  a  key  driver  for  the  scope  of  these  validation  steps  and  we  have 
created  a  model  for  this  effort.  In  addition,  Einstone  and  Turoff  recommend  asking  a 
series  of  questions  to  evaluate  whether  the  Delphi  Method  is  a  desirable  choice  for  an 
information  gathering  process.  These  questions  are:  (1)  Does  the  issue  not  require  a 
precise  analytical  technique,  but  can  be  evaluated  by  subjective  judgment?  (2)  Do  the 
individuals  need  to  contribute  to  the  examination  of  a  broad  or  complex  problem  have  no 
history  of  adequate  communication  and  may  represent  a  diverse  background  with  respect 
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to  experience  and  expertise?  (3)  Can  the  diversity  and  independence  of  the  subject  matter 
experts  (SMEs)  be  preserved  to  assure  validity  of  the  results  and  avoid  group  think 
[Linstone  &  Turoff,  2002]?  Because  the  validation  of  the  compatibility  of  R&D  satellites 
and  their  ground  systems  does  not  have  a  documented  process  and  is  done  differently  for 
every  program,  we  feel  that  this  issue  can  be  evaluated  by  subjective  judgment.  We  also 
feel  that  in  order  to  ensure  that  all  stakeholders  were  represented,  our  SMEs  needed  to 
have  a  variety  of  experiences  and  expertise.  Einally  to  address  the  last  question,  the 
primary  reason  we  choose  to  collect  our  data  using  the  Delphi  Method  was  to  avoid 
group  think.  Eor  these  reasons  we  feel  that  the  Delphi  Method  was  the  appropriate  choice 
for  conducting  the  research  for  our  thesis. 

3.2.2  Selection  of  Subject  Matter  Experts 

The  panel  of  experts  used  for  our  thesis  comes  from  a  variety  of  different 
backgrounds.  According  to  Dean,  Wood,  Moore,  and  Bogart  this  helps  to  avoid  the  three 
sources  of  error  introduced  by  experts.  These  experts  weighed  in  on  whether  or  not  we 
built  the  correct  model,  and  determined  whether  the  model  yields  accurate  cost  data. 

In  order  to  create  our  panel  of  SMEs,  we  looked  within  our  organization,  the  Space 
Development  and  Test  Wing  (SDTW),  to  establish  a  wide  range  of  panel  members  with 
various  areas  of  expertise.  We  assembled  a  panel  of  eight  experts  for  this  thesis.  Our 
SMEs  consisted  of  ourselves,  a  military  member  that  works  for  the  organization  that 
provides  test  assets  to  the  Space  Community,  a  government  civilian,  two  ground  system 
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contractors,  an  operations  contractor,  and  an  individual  from  our  independent  technical 
advisory  contract. 

The  ground  system  contractor  is  responsible  for  the  procurement  and  development  of 
new  ground  systems  and  sustainment  of  both  pre-existing  and  new  ground  systems.  They 
are  a  key  team  member  during  many  stages  of  the  validation  process  and  therefore  have 
insight  into  both  the  process  and  the  ramifications  of  ground  system  risk  realizations. 

The  operations  contractor  personnel  operate  the  satellites.  They  are  members  of  each 
integrated  product  team  throughout  the  entire  mission  life  cycle.  The  operations 
personnel  have  expert  knowledge  about  the  ramifications  of  risk  realization  pertaining  to 
both  the  satellite  and  the  ground  system. 

The  independent  technical  advisors  are  the  team  of  individuals  that  our  commanders 
turn  to  for  technical  consultation.  They  have  a  wide  breadth  of  experience.  They  gave  an 
objective  perspective  to  our  thesis.  The  specific  independent  technical  advisor  on  our 
panel  has  expertise  in  both  ground  system  development  and  satellite  operations. 

The  military  and  government  civilians  were  selected  based  on  their  experiences 
within  the  wing,  and  their  unique  perspective  based  on  the  programs  they  had  worked  and 
the  years  of  experience  they  brought  to  the  table.  In  the  following  paragraphs  we  discuss 
the  unique  expertise  of  each  SME.  In  order  to  keep  our  SMEs  identities  confidential, 
excluding  ourselves,  we  have  given  each  SME  a  number. 

We  each  acted  as  a  SME  for  this  thesis.  SME  #1,  Mary  Trautwein,  is  a  E‘  Et  in  the 
United  States  Air  Eorce.  She  has  worked  at  SDTW  for  over  three  years.  She  has  been  an 
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On-Orbit  Mission  Lead,  a  Mission  Ground  System  and  Satellite  Test  Lead  and  a  Mission 
Ground  System  Development  Lead.  All  of  these  missions  flew,  or  will  fly,  out  of 
SDTW. 

SME  #2,  Amanda  Langenbrunner,  is  a  Captain  in  the  United  States  Air  Foree.  She 
has  worked  at  SDTW  for  over  three  years.  She  has  been  the  Operations  Flight 
Commander  for  the  squadron  within  SDTW  that  operates  R&D  satellites.  She  is 
eurrently  working  for  the  Department  of  Defense  (DoD)  Spaee  Test  Program  (STP) 
manifesting  R&D  satellite  missions. 

SME  #3  is  a  Captain  in  the  United  States  Air  Foree.  He  has  worked  at  SDTW  for  one 
year.  He  is  the  Mobile  Range  Flight  Commander.  This  is  the  organization  that  provides 
test  assets  to  the  Spaee  Community.  SME  #3  was  ehosen  as  a  part  of  our  panel  of 
experts,  due  to  his  position  as  the  Mobile  Range  Flight  Commander.  He  is  responsible 
for  eustomer  serviee  and  all  eost  and  eontraeting  aetions  for  the  test  assets  provided  by 
the  Mobile  Range  Flight.  He  is  eurrently  the  SDTW  expert  on  availability,  eost,  and 
operations  of  these  important  test  assets. 

SME  #4  is  eurrently  a  government  eivilian  with  the  Responsive  Satellite  Command 
and  Control  Division  at  SDTW.  He  has  worked  at  SDTW  for  9  years.  He  is  the  Chief 
Architeet  for  the  development  of  a  new  ground  system  that  will  fly  satellites  at  SDTW 
and  the  50*  Spaee  Wing  in  Colorado  Springs. 

SME  #5  eurrently  works  for  the  Operations  Contraet  at  SDTW.  He  has  worked  at  the 
Spaee  Development  and  Test  Wing  for  8.5  years.  He  is  eurrently  the  Operations  Mission 


25 


Lead  for  a  satellite  mission  flying  at  SDTW.  He  has  more  than  12  years  of  experience  in 
the  Space  Industry. 

SME  #6  currently  works  for  the  Ground  System  Development  Contract  in  the  RSC. 
He  has  worked  at  SDTW  for  more  than  10  years.  He  is  currently  the  technical  lead  for 
the  ground  system  development  of  a  satellite  mission  that  will  fly  at  SDTW  within  the 
next  12  months.  In  the  past  he  has  worked  as  the  Ground  System  Development  Lead  for 
a  past  ground  system,  and  the  overall  hardware  architect  for  the  newest  Ground  System  to 
be  used  at  SDTW. 

SME  #7  currently  works  for  the  Ground  System  Development  Contract  at  SDTW. 

His  current  position  is  Project  Lead  and  Hardware  Systems  Engineer.  He  has  held  this 
position  for  the  last  12  years.  Prior  to  working  at  SDTW  he  was  the  Senior  Hardware 
Engineer  with  Loral  Space  and  Range  Systems  in  Sunnyvale,  CA.  SME  #7  has  37.5 
years  experience  with  the  Air  Eorce  Satellite  Control  Network  (AESCN). 

SME  #8  is  a  senior  technical  advisor  for  the  government  at  SDTW.  He  has  worked  at 
SDTW  for  16  years.  He  has  worked  with  satellites  at  SDTW  for  16  years  and  ground 
systems  at  SDTW  for  12  years.  SME  #8  has  worked  in  the  space  industry  for  over  30 
years. 

Overall,  our  panel  of  experts  has  a  long  history  of  experience  in  all  aspects  of  the 
R&D  satellite  business. 
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3.2.3  Development  of  the  Delphi  Survey 

In  order  to  develop  our  validation  model  and  assign  appropriate  costs  to  each  activity, 
we  conducted  research  using  the  Delphi  Method.  We  issued  a  series  of  surveys  to  a 
group  of  SMEs.  All  of  the  questions  in  these  surveys  had  answers  that  required  the 
SMEs  to  select  an  answer  that  was  either  a  binary  (i.e.  Yes,  No  or  Agree,  Disagree)  or 
scaled  (i.e.  Negligible,  Minor,  Moderate,  Serious  or  Critical)  answer.  The  SMEs  also  had 
the  option  of  providing  written  explanations  of  their  answers.  The  surveys  were 
conducted  this  way  to  limit  the  SME  answers  in  order  to  obtain  a  consensus.  However, 
the  surveys  also  allowed  the  SMEs  to  explain  their  opinions  so  that  other  SMEs  could 
either  accept  or  dispute  them.  In  the  first  survey  we  also  asked  open  ended  questions. 
These  questions  and  responses  were  considered  when  we  created  the  second  survey  and 
were  used  in  the  final  discussion. 

In  the  first  survey,  we  defined  the  Initial  Validation  Process  using  the  Canonical 
Verification,  Validation  and  Test  (VVT)  Model  (CVM)  shown  in  Appendix  A.  This 
process  was  developed  based  on  our  combined  years  of  ground  system  and  satellite 
compatibility  validation  testing.  Throughout  this  time,  we  have  been  involved  in  the 
validation  testing  for  six  satellite  missions.  Each  mission  had  a  unique  validation  plan, 
and  therefore  unique  CVMs.  We  ensured  that  all  possible  validation  steps  were  included 
in  the  plan.  We  gave  the  CVM  to  the  SMEs  to  analyze  the  process  and  provide 
comments  to  yield  a  correct  model.  Next,  we  asked  our  SMEs  to  define  the  cost  of 
executing  each  step  in  the  process.  Additionally,  the  SMEs  defined  risks  associated  with 
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failing  to  complete  various  validation  efforts.  They  defined  these  risks  based  on  the 
Strategy  Based  Risk  Model  (SRM).  First  they  defined  the  risk  attributes  and  then  the  two 
risk  variables  were  derived.  The  two  risk  variables  are  the  probability  that  the  risk  will 
impact  the  mission,  and  the  severity  of  that  impact.  The  results  from  Survey  #1  were  the 
basis  for  the  creation  of  Survey  #2. 

During  Survey  #1,  we  did  not  get  any  cost  data  from  several  SMEs.  Upon  further 
inquiry  with  the  SMEs  we  were  informed  that  they  did  not  have  the  time  to  collect  this 
information  and  did  not  want  to  give  us  incorrect  data.  We  instead  collected  the 
information  ourselves  and  presented  the  cost  data  to  them  in  Survey  #2  for  comment.  To 
do  this,  we  created  point  estimates  by  looking  at  each  contract  for  each  validation 
activity.  Eor  every  program,  we  always  have  a  ground  system  contract,  an  operations 
contract,  a  satellite  development  contractor  and  the  organization  that  provides  test  assets 
to  the  Space  Community.  We  looked  at  each  of  these  contracts  and  collected  cost  data 
from  four  programs.  These  programs  varied  in  budget,  mission  and  launch  date.  We 
looked  at  programs  that  have  launched  or  will  launch  between  2001  and  2010.  Some  of 
these  programs  had  actual  cost  data  and  hours  associated  with  steps  in  the  validation 
process.  Others  only  had  contractor  proposals  because  either  the  validation  steps  have  yet 
to  be  conducted,  or  actual  cost  data  was  not  recorded.  In  order  to  account  for  inflation 
between  2001  and  2009,  we  examine  the  number  of  hours  either  executed  or  proposed  for 
the  validation  steps  and  applied  a  current  hourly  rate  for  each  organization.  The  test 
asset  organization  provided  a  menu  of  pre-defined  prices  for  use  of  each  of  their  test 
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assets;  we  used  this  to  identify  the  costs  for  use  of  the  test  assets.  To  do  this,  we  had  to 
remove  programmatic  anomalies  to  get  an  estimate  for  the  cost  of  each  activity.  Some  of 
the  anomalies  that  we  call  “outliers”  are  listed  below: 

•  A  program  had  a  7  year  slip  that  included  the  program  being  cancelled  and 
resumed.  This  program  had  three  Factory  Compatibility  Tests  (FCT),  we  only 
used  the  cost  data  on  the  final  FCT. 

•  Due  to  the  complexity  of  one  program  and  the  high  level  interest,  the  program 
completed  command  and  telemetry  validation  on  every  single  command  that 
could  be  passed  from  the  ground  system  to  the  satellite  multiple  times.  We  chose 
to  look  at  the  cost  data  from  one  round  of  command  and  telemetry  validation  as 
this  is  the  preferred  method  of  command  and  telemetry  validation  for  programs. 

•  One  program  chose  to  do  as  little  validation  as  possible  due  to  schedule  and 
budget.  This  program  knew  they  were  putting  a  satellite  in  orbit  with  a  large 
amount  of  risk.  We  only  looked  at  cost  data  from  this  program  on  validation  steps 
they  performed. 

After  eliminating  the  outliers  and  deriving  point  estimates  for  each  validation  activity, 
we  added  these  point  estimates  to  Survey  #2  for  comments  and  feedback  from  our  SMEs. 

In  Survey  #2,  all  of  the  questions  asked  in  Survey  #1  were  included  and  the  SMEs 
were  asked  to  agree  or  disagree  with  the  other  SMEs.  Additionally  all  comments  from 
Survey  #lwere  incorporated.  The  SMEs  were  also  asked  to  agree  or  disagree  with  each 
of  these  comments.  We  added  the  point  estimates  calculated  after  Survey  #1  into  Survey 
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#2  and  asked  the  SMEs  whether  these  estimates  appeared  too  high,  too  low,  or  correct. 
We  also  asked  them  to  comment  on  why  they  felt  this  way.  Finally,  we  left  additional 
space  on  the  survey  for  any  additional  comments.  The  feedback  received  from  Survey  #2 
was  used  to  create  Survey  #3. 

In  the  formation  of  Survey  #3,  any  question  from  Survey  #2  that  everyone  agreed  on, 
was  considered  truth,  and  it  was  left  off  Survey  #3.  We  included  all  of  the  disparities 
from  Survey  #2,  we  asked  the  SMEs  once  again  to  agree  or  disagree  with  these 
comments.  We  then  asked  the  SMEs  specific  questions  about  each  of  the  areas  of 
disparity  to  try  and  understand  the  rationale.  All  of  the  SMEs  concurred  on  each  of  the 
risks  and  its  attributes.  As  a  result.  Survey  #3,  focused  on  refining  the  risk  variables  and 
steps  in  the  validation  process  that  mitigated  the  risk. 

Through  these  surveys,  the  group  was  given  the  opportunity  to  comment  on  the 
responses  of  other  SMEs,  while  simultaneously  allowing  them  to  change  their  responses 
as  a  result  of  reviewing  others  answers  and  explanations.  The  iterations  were  complete 
when  there  was  a  final  group  agreement  and  when  we  believed  the  experts  were  no 
longer  changing  their  opinion  or  commenting  on  the  opinions  of  the  other  panel 
members.  This  is  defined  as  saturation  [Rayens  and  Hahn,  2000].  For  the  scope  of  this 
thesis  saturation  was  reached  when  each  SMEs  response  changed  less  than  5%.  The  5% 
was  determined  by  adding  up  the  number  of  questions,  and  subset  of  questions  in  Survey 
#2.  Survey  #2  was  used  because  it  had  all  18  risks  identified  in  the  survey,  it  included  the 
cost  data  for  the  validation  steps,  and  it  did  not  include  the  open  ended  questions  asked  in 


30 


Survey  #1.  The  total  number  of  questions  in  Survey  #2  is  148,  5%  of  this  is  7.4.  From 
this  we  conclude  that  saturation  was  reached  when  each  SME  changed  their  answer  on 
less  than  7  questions.  This  occurred  on  Survey  #3. 

When  receiving  data  from  experts  we  documented  a  wide  range  of  responses.  As  the 
iterations  completed,  the  original  spectrum  of  responses  was  narrowed.  The  spectrum 
narrowed  after  the  second  survey  and  reached  saturation  following  the  third  survey. 

3.2.4  Administration  of  the  Survey 

The  surveys  were  distributed  through  electronic  mail,  with  each  SME  as  a  blind 
courtesy  copy.  This  ensured  that  each  SME  received  the  same  instruction  and  survey, 
while  ensuring  the  integrity  of  the  system.  A  systematic  procedure  allows  the  experts  to 
have  a  sense  of  objectivity  throughout  the  study  [Dalkey,  1969].  The  group  of  experts 
only  interacted  with  one  another  through  the  feedback  loop  that  was  established.  The 
SMEs  responded  in  one  of  two  methods.  Either  they  completed  the  surveys  in  hard  copy 
and  delivered  them  back  to  us  or  they  filled  out  a  soft  copy  of  the  survey  and  emailed  it 
back  to  us.  To  ensure  we  were  not  swayed  by  the  beliefs  of  the  other  SMEs  we 
completed  our  surveys  immediately  after  we  sent  them  out,  and  thus  before  we  received 
any  feedback  from  the  other  SMEs. 

3.3  Methodology  for  Modeling  VVT  Risks  and  Costs 

As  Engel  and  Barad  used  a  four  step  methodology  to  model  their  VVT  risks  and  costs, 
we  will  also  used  four  steps.  The  details  for  each  step  in  our  methodology  are  explained 
in  the  subsequent  sections. 


31 


3.3.1  Define  the  Validation  Process 


The  first  step  in  analyzing  the  costs  vs.  risks  associated  with  the  validation  of  the 
compatibility  of  a  satellite  and  its  ground  system  is  to  define  the  process.  In  order  to  do 
this,  we  developed  a  CVM  of  all  steps  associated  with  this  process.  The  CVM  discussed 
in  chapter  two  was  used  when  developing  the  model.  Each  node  of  the  diagram  is  a 
discrete  validation  step.  We  developed  the  first  iteration  of  this  process,  shown  in 
Appendix  A,  from  satellite  programs  flown  by  the  RSC.  This  process  was  developed 
based  on  our  combined  experience  in  ground  system  and  satellite  compatibility  validation 
testing.  Throughout  this  time,  we  have  been  involved  in  the  validation  testing  for  six 
different  satellite  missions.  As  each  mission  had  a  unique  validation  plan,  they  had 
unique  CVMs.  We  took  the  base  validation  plan  and  ensured  we  incorporated  all 
possible  validation  steps  into  the  plan.  The  process  was  provided  to  our  SMEs  for 
evaluation.  The  SMEs  provided  feedback  through  the  Delphi  Method.  A  final  CVM  was 
created  from  this  feedback. 

3.3.2  Assign  Appropriate  Cost  to  Each  Step 

The  value  of  a  validation  step  has  two  parts;  the  first  is  defined  as  the  cost  of  the 
validation  step.  The  costs  associated  with  each  step  in  the  validation  process  were  based 
on  hours  of  work  required  to  complete  the  task.  They  were  calculated  using  actual  cost 
data  and  contractor  estimates.  The  costs  were  included  in  the  surveys  provided  to  our 
SMEs  and  changes  were  made  based  on  SME  feedback. 
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In  order  to  define  the  costs  of  the  validation  steps  we  used  expert  opinion.  According 
to  Dean,  Wood,  Moore  and  Bogart  in  Cost  Risk  Analysis  Based  on  Perception  of  the 
Engineering  Process  many  cost  risk  analyses  are  based  upon  an  expert’s  knowledge  of 
the  cost  of  similar  projects  in  the  past  [Dean,  Wood,  Moore  &  Bogart,  1986].  They 
applied  this  method  by  asking  managers  and  engineers  to  estimate  the  cost  of  a  project  in 
their  area  of  expertise  based  on  historical  data  or  similar  projects.  This  is  an  excellent 
method  for  estimating  costs,  however  according  to  Dean,  Wood,  Moore,  and  Bogart  there 
are  three  sources  of  error  that  are  introduced  by  using  expert  opinion.  The  first  is  that  the 
historical  cost  data  may  be  in  error  by  some  unknown  amount.  For  example  inflation 
needs  to  be  considered.  Also  the  application  of  the  task  may  be  different  or  modernized 
equipment  could  be  available.  The  second  source  of  error  is  that  the  expert  may 
inaccurately  evaluate  the  new  project’s  similarities  to  an  older  project  and  provide 
inaccurate  estimations  based  on  this.  The  third  source  of  error  is  that  the  factors  used  to 
adjust  the  costs  of  an  old  project  may  not  correctly  reflect  the  new  project.  In  order  to 
reduce  the  error  caused  by  these  three  sources  Dean,  Wood,  Moore,  and  Bogart  used  a 
range  of  cost  estimations.  This  method  allows  for  a  higher  level  of  confidence  in  the 
accuracy  of  the  expert  estimations  [Dean,  Wood,  Moore  &  Bogart,  1986]. 

In  order  to  help  eliminate  the  sources  of  error  introduced  by  Dean,  Wood,  Moore  and 
Bogart,  we  went  through  several  programs  and  came  up  with  a  point  estimate  for  each 
validation  activity.  The  derivation  of  these  point  estimates  was  based  on  the  hours  of 
work  required  to  complete  the  task.  They  were  calculated  using  actual  cost  data  and 
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contractor  estimates.  The  process  for  finding  these  point  estimates  is  explained  in  further 
detail  in  Section  3.2.3.  These  point  estimates  were  given  to  the  SMEs  to  determine  if 
they  believed  the  cost  of  each  validation  effort  was  too  high,  too  low,  or  acceptable. 

They  then  gave  us  rationale  for  their  beliefs. 

The  second  cost  associated  with  the  value  of  a  validation  step  is  the  impact  cost  of  the 
risk  being  realized.  During  Step  2  of  our  methodology,  we  identified  program  risks  that 
can  be  mitigated  by  executing  steps  in  the  validation  process,  and  we  defined  costs 
associated  with  the  impacts  of  the  risks. 

3.3.3  Identify  Program  Risks 

Like  Engel  and  Barad  we  used  the  Strategy  based  Risk  Model  (SRM)  to  define  risks 
associated  with  the  compatibility  between  a  satellite  and  its  ground  system.  We  defined 
risk  attributes  (risk  description  and  validation  steps  associated  with  this  risk)  and  risk 
variables  (probability  and  severity  of  the  risk  impacts).  However  unlike  Engel  and 
Barad,  this  thesis  will  not  use  the  probabilities  and  impact  levels  identified  in  MIL-STD- 
882C.  Instead  we  will  use  the  risk  chart  that  is  used  and  accepted  within  SDTW.  This 
chart  is  based  on  the  MIL-STD-882C  but  is  tailored  for  R&D  satellite  programs.  The 
probabilities  used  and  SDTW  are:  0-10%,  11-40%,  41-60%,  61-90%  and  91-100%.  The 
Severity  levels  used  at  SDTW  are:  Critical,  Serious,  Moderate,  Minor  and  Negligible. 
These  severity  levels  are  more  qualitatively  defined  than  in  the  MIL-STD-882C.  This  is 
for  several  reasons.  Budgets  can  vary  from  mission  to  mission,  money  means  different 
things  to  each  mission.  Eor  example  $1M  is  worth  a  lot  more  to  a  program  office  with  a 
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$30M  budget  than  it  is  to  a  program  office  with  a  $200M  budget.  The  second  reason  is 
that  severity  of  risk  may  not  be  tied  to  the  cost  of  resolving  the  risk.  The  cost  of 
resolving  the  risk  may  only  be  $20,000  and  one  week,  but  if  it  is  not  resolved  realization 
of  this  risk  could  lead  to  a  loss  of  mission  and  therefore  would  still  be  carried  as  critical. 

The  risk  variables  (the  probability  and  severity)  were  defined  for  each  risk.  The 
probability  was  defined  twice.  First  the  probability  was  defined  based  on  the  risk  prior  to 
performing  any  mitigation  steps.  Then,  the  probability  was  defined  after  the  execution  of 
the  steps  in  the  validation  process.  This  is  an  improvement  over  Engel  and  Barad’s 
model.  Engle  and  Barad  assumed  that  if  steps  were  taken  to  mitigate  the  risk,  then  the 
risk  would  not  occur.  We  acknowledged  that  this  is  not  realistic  and  this  is  why  we 
calculate  the  probability  of  the  risk  being  realized  before  and  after  the  validation  process 
is  executed.  The  validation  steps  mitigate  the  risk  and  thus  the  probability  of  the  risk 
being  realized  is  less,  but  it  is  not  zero  [Engel  &  Barad,  2003].  Once  the  risks  were 
determined  and  fully  defined,  there  are  multiple  options  for  mitigating  risks.  These 
include:  accept,  avoid,  reduce,  share  and  transfer.  Risk  avoidance  is  a  response  to  a  threat 
that  eliminates  its  probability  of  impact  on  the  project.  Risk  transfer  is  a  response  to  a 
threat  that  transfers  the  risk  to  a  third  party  who  is  better  able  to  manage  the  risk.  Risk 
reduction  is  a  response  to  a  threat  that  reduces  its  probability  and/or  impact  on  the 
project,  aiming  to  reduce  the  risk  to  an  acceptable  level.  Einally  risk  acceptance  is  a 
response  where  either  no  proactive  action  is  taken  or  where  responses  are  designed  that 
are  contingent  upon  a  change  in  circumstances  [Hillson  &  Simon,  2007].  Eor  the  scope 
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of  this  thesis  we  will  only  be  focusing  on  reducing  risks  through  the  validation  process, 
and  accepting  them  due  to  budgetary  constraints  that  prevent  the  program  from  executing 
every  step  in  the  validation  process.  If  the  risk  can  be  reduced,  the  program  office  must 
evaluate  the  cost  associated  with  the  reduction  efforts.  These  costs  were  defined  as  part 
of  our  thesis.  This  helps  determine  whether  the  program  manager  should  mitigate  the 
risk,  or  assume  the  risk  and  plan  for  realization.  In  order  to  do  this,  cost  drivers  were 
modeled.  The  validation  process  was  designed  to  mitigate  a  set  of  known  risks  associated 
with  the  compatibility  between  an  R&D  satellite  and  its  unique  ground  system.  Through 
our  panel  of  experts,  18  program  risks  were  identified.  For  each  of  these  risks,  we 
ascertained  a  definition,  probability  of  impact,  and  cost  of  impact.  This  allowed  us  to 
assign  a  value  to  each  of  the  validation  steps.  In  order  to  validate  that  this  list  of  risks 
was  in  fact  a  complete  and  correct  list  we  investigated  past  R&D  satellite  programs.  We 
were  able  to  locate  risk  registers  for  five  satellite  programs.  Each  program  identified 
between  10  and  16  risks  associated  with  the  compatibility  between  the  satellite  and  the 
ground  system.  Each  one  of  the  risks  identified  on  the  risk  registers  was  a  risk  that  we 
identified  in  our  thesis.  Although  this  number  is  less  than  the  18  we  identified,  our  list  of 
risks  was  complete.  Some  of  the  programs  combined  similar  risks.  Occasionally  due  to 
the  unique  nature  of  the  R&D  satellites  and  their  ground  systems,  sometimes  a  risk  was 
not  present.  The  probability  of  realizing  the  risk  is  lower  if  the  step  in  the  validation 
process  is  executed.  We  asked  our  SMEs  to  identify  the  probability  of  the  risk  being 
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realized  before  the  validation  effort,  and  after.  This  allowed  us  to  define  the  importance 
of  the  associated  validation  steps. 

3.3.4  Define  Costs  Associated  with  the  Impacts  of  Risks 

It  is  important  to  characterize  the  costs  associated  with  the  impacts  of  each  of  the  risks 
identified.  A  cost  value  was  assigned  to  the  impact  of  each  risk.  The  cost  value  was 
defined  through  the  Delphi  Method.  The  SMEs  identified  the  cost  impacts  in  Survey  #1 
and  in  subsequent  surveys,  other  SMEs  agreed  or  disagreed  with  the  cost  of  the  risk 
impact.  When  there  was  disagreement,  we  asked  the  SMEs  specifically  why  they 
disagreed  and  then  added  their  comments  to  the  next  iteration  of  the  survey  for  agreement 
from  the  other  SMEs.  However,  during  the  original  process,  we  didn’t  ask  the  SMEs  to 
identify  why  they  disagreed.  As  a  result,  we  had  to  send  follow-up  emails  to  the  SMEs 
that  disagreed  after  Survey  #2,  so  that  we  could  include  their  comments  in  Survey  #3.  If 
the  risk  was  a  schedule  slip  or  technical  risk  the  cost  value  assigned  correlated  with  the 
associated  schedule  slip,  and/or  the  cost  of  resolving  the  technical  issue. 

3.3.5  Transform  the  Validation  Model  into  a  Simulation 

Monte  Carlo  simulation  uses  random  number  generation  to  simulate  the  occurrence  or 
nonoccurrence  of  a  given  probabilistic  event  according  to  given  distributions  [Engel  & 
Barad,  2003].  In  our  simulation,  the  probabilistic  event  is  the  realization  of  a  risk 
associated  with  the  compatibility  between  a  satellite  and  its  ground  system.  Erom  this 
many  hypothetical  scenarios  of  risk  impacts  may  be  generated.  These  scenarios  were 
used  create  distributions  of  overall  validation  costs.  Our  simulation  follows  the  process 
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identified  by  Engel  and  Barad  and  detailed  below.  First  we  defined  the  validation  effort 
using  the  CVM  and  calculated  the  deterministic  costs  of  the  performed  validation  steps. 
Then  we  simulated  the  risks  impact  costs  stemming  from  the  validation  steps  that  were 
not  performed.  Finally  we  summed  the  overall  deterministic  validation  costs  and  the  risk 
impact  costs  [Engel  &  Barad,  2003]. 

The  cost  of  a  particular  validation  strategy  is  deterministic  and  does  not  change  from 
run  to  run.  The  validation  steps  that  will  be  performed  were  determined  for  a  given 
strategy  and  defined  up  front.  The  overall  cost  of  a  given  validation  strategy  was 
calculated  using  Equation  1  below.  The  decision  variable  is  X.  If  a  step  in  the  validation 
process  was  fully  performed  X  =  1,  if  a  step  in  the  validation  process  was  not  performed 
X  =  0.  This  is  determined  by  the  validation  strategy  [Engel  &  Barad,  2003]. 

^Validation  _strategy  =  YiCn*X) 

Cvaiidation  .strategy  =  Overall  cost  of  Validation  strategy 
Q  =  Costs  associated  with  executing  validtion  step 
X  =  0,1  as  determined  by  validation  strategy 

[Engel  &  Barad,  2003] 

Risk  impact  costs  are  probabilistic  and  must  be  generated  using  a  Monte  Carlo 
simulation  and  random  number  generation.  During  each  simulation  run,  a  random 
number  was  generated  for  each  risk  to  determine  an  occurrence  or  a  nonoccurrence  of 
risk  impact  cost,  according  to  its  respective  given  probability.  The  simulation  run  was  a 
decision  point  for  each  risk  to  determine  if  the  risk  was  realized.  If  the  risk  was  realized. 
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the  associated  impact  costs  were  applied  for  that  risk.  If  the  risk  was  not  realized  then  the 
associated  impact  costs  were  zero.  For  each  simulation  run,  the  impact  costs  for  the 
validation  strategy  were  calculated  using  the  equation  below  [Engel  &  Barad,  2003]. 


^Impact  _strategy  Impact  _riskl’  ^Impact  _risk  2  ■■■  ^Impact  _riskn 


) 


(Equation  2) 

hmpac  tstrategy  ~  Impuct  Costs  JOT  u  giveu  simulation  run 

[Engel  &  Barad,  2003] 

Eor  each  Monte  Carlo  simulation  run  the  overall  validation  costs  incurred  were: 


-  _  j  ,  -  (Equation  3) 

^overall  _validation  _costs  hmpact  _strategy  '  ^validation  _strategy 

(^overall  _vaidation  _costs  =  Overall  Validation  Costs  for  a  given  simulation  run 

[Engel  &  Barad,  2003] 

This  simulation  was  run  a  number  of  times  to  perform  a  trial.  When  applied  to  the 
sample  program  in  chapter  four  of  this  thesis,  a  trial  consisted  of  1,000  runs  of  the 
simulation.  The  results.  Overall  Validation  Costs,  for  every  simulation  trial  were 
depicted  in  a  histogram.  Eigure  6  is  an  example  histogram  for  one  trial  of  a  1,000 
simulation  runs. 
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Example  1 


Figure  6:  Example  Histogram  of  Simulation  Trial 

In  this  figure  the  Y-axis  depicts  the  percentage  of  runs  that  yielded  an  overall 
validation  cost  value  corresponding  to  the  ones  shown  on  the  X-axis.  From  this 
histogram  it  is  useful  to  calculate  the  following  information  for  comparison  to  simulation 
trials  of  other  validation  strategies:  the  range,  the  mean,  the  standard  deviation,  and  the 
median.  These  data  points  allow  program  offices  to  directly  compare  multiple  validation 
strategies  and  decipher  which  is  the  most  effective  based  on  their  budget  and  adversity  to 
risk. 
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3.3.6  Evaluate  the  Simulation 


The  simulation  created  in  Step  3  of  our  methodology  will  be  available  to  SDTW.  It 
will  provide  them  with  the  ability  to  evaluate  potential  validation  plans,  thereby  focusing 
resources  on  the  validation  steps  with  the  most  value.  R&D  satellite  program  offices  will 
be  able  to  use  this  technique  to  have  a  quantitative  method  for  determining  validation 
efforts  based  on  cost  and  risk. 

The  fourth  and  final  step  in  our  methodology  evaluated  the  accuracy  of  our  simulation. 
We  accomplished  this  in  two  ways.  We  first  applied  our  simulation  to  a  sample  program 
that  is  performing  validation  steps  in  preparation  for  operations  in  the  RSC.  We 
compared  three  different  validation  strategies  to  help  evaluate  the  accuracy  and  usability 
of  our  simulation. 

Finally,  in  order  to  demonstrate  the  accuracy  of  the  simulation  used  to  generate 
distributions  of  possible  costs  and  risk  outcomes,  we  provided  the  simulation  to  two 
program  managers.  We  asked  the  program  managers  to  run  the  simulation  for  their  on¬ 
going  programs  and  answer  a  series  of  questions.  We  asked  them  to  also  keep  in  mind 
past  programs  when  answering  the  questions.  The  first  set  of  question  we  asked  dealt 
with  the  risks  identified  by  the  Delphi  Method.  We  first  asked  the  program  managers  if 
these  risks  encompassed  all  the  risks  on  their  current  satellite  program.  Next  we  asked 
both  program  managers  if  they  had  ever  worked  an  R&D  satellite  program  that  tracked  a 
risk  not  identified  in  our  thesis.  The  next  set  of  questions  we  asked  the  program 
managers  dealt  with  their  thoughts  on  the  simulation  technique.  Specifically,  what  were 
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the  results  using  the  simulation  on  their  program?  Does  using  the  simulation  help  save 
their  programs  cost  or  schedule?  In  addition,  how  could  utilizing  the  simulation  help 
their  program? 
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IV.  Analysis  and  Results 


4.1  Introduction 

Throughout  this  chapter  we  will  be  examining  the  results  we  obtained  through  the 
Delphi  Method.  We  will  be  demonstrating  how  we  used  these  results  to  create  our 
simulation.  We  will  apply  these  results  to  a  sample  Research  and  Developmental  (R&D) 
satellite  program  and  present  program  manager  feedback  to  demonstrate  the  accuracy  of 
our  findings.  Initially  we  started  with  a  draft  of  the  validation  process,  which  we 
provided  to  our  Subject  Matter  Experts  (SMEs)  using  the  Delphi  Method.  Through  their 
feedback,  we  identified  the  Einal  Validation  Process  and  costs  associated  with  executing 
each  step.  Through  the  Delphi  Method  our  SMEs  also  provided  the  risks  associated  with 
the  compatibility  of  a  satellite  and  its  ground  system  and  what  specific  steps  in  the 
validation  process  mitigated  these  risks.  Eor  each  risk  we  asked  our  SMEs  to  define  the 
probability  of  impact,  severity  of  impact,  and  impact  costs.  Erom  this  information  we 
were  able  to  create  a  simulation  that  generates  distributions  of  possible  risk  and  cost 
outcomes.  This  information  will  be  used  to  help  R&D  satellite  program  offices  evaluate 
the  fidelity  of  their  proposed  validation  strategy.  They  can  use  our  simulation  to  compare 
validation  strategies  and  assess  if  their  validation  strategy  is  complete  or  if  there  are  steps 
that  can  be  skipped  to  preserve  cost.  This  section  of  our  thesis  will  present  these 
findings,  discuss  our  SME  feedback,  apply  our  simulation,  and  demonstrate  the  fidelity  of 
this  method. 
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4.2  The  CVM  for  R&D  Satellite  Validation 


We  developed  the  initial  Canonical  Validation  Model  (CVM)  based  on  our  combined 
experience  in  ground  system  and  satellite  compatibility  validation  testing.  This  was 
provided  to  our  SMEs  for  evaluation.  The  final  CVM  was  created  based  on  the  SME 
feedback  through  the  Delphi  process.  In  the  sections  below  we  will  present  the  initial  and 
final  CVMs. 

4.2.1  The  Initial  CVM 

Shown  below,  Eigure  7  depicts  our  initial  draft  of  the  validation  process.  During  the 
execution  of  the  Delphi  Method,  we  provided  the  draft  CVM  to  our  SMEs  as  a  starting 
point  for  their  comments.  A  detailed  description  of  this  process  can  be  found  in 
Appendix  A.  This  process  was  developed  based  on  our  combined  experience  in  ground 
system  and  satellite  compatibility  validation  testing  within  the  Research  Development 
Test  and  Evaluation  (RDT&E)  Support  Complex  (RSC).  Throughout  this  time,  we  have 
been  involved  in  the  validation  testing  for  six  satellite  missions.  As  each  mission  had  a 
unique  validation  plan,  they  had  unique  CVMs.  We  incorporated  all  possible  validation 
steps  into  the  Initial  Validation  Process. 
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0.0:  Validation  Process 


Figure  7:  Initial  Validation  Process 


4.2.2  Final  CVM 

Shown  below,  Figure  8  depicts  the  Final  Validation  Process.  This  process  was 
developed  based  on  feedback  ascertained  throughout  the  Delphi  Method.  Changes  were 
made  from  the  Initial  Validation  Process.  We  deleted  Mission  Dress  Rehearsal  because 
typically  the  ground  system  does  not  change  after  Launch  Based  Compatibility  Test 
(LBCT)  and  is  “frozen”  prior  to  this  event,  therefore  is  not  a  part  of  the  Validation 
Process.  Also  Exercises  and  Rehearsals  were  condensed.  Rather  than  calling  out  each 
Exercise  and  Rehearsal  individually,  one  block  is  shown  for  Exercises  and  one  block  for 
Rehearsals.  This  was  done  because  Exercises  and  Rehearsals  are  primarily  training 
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events,  but  they  have  a  secondary  mission  of  testing  the  ground  system  and  operational 
concept  under  realistic  loading  conditions.  Also  Exercises  and  Rehearsals  are  often 
executed  as  needed  throughout  the  Validation  Process.  Our  SMEs  concluded  from  this 
that  they  did  not  need  to  be  individually  called  out  in  our  process.  The  order  of  the 
validation  process  was  debated  throughout  the  surveys.  Einally  all  SMEs  concurred  that 
the  order  of  the  validation  process  will  vary  from  mission  to  mission,  particularly  the 
placement  of  Exercises  and  Rehearsals.  All  of  the  SMEs  agreed  the  order  of  our  Einal 
Validation  Process  represented  a  typical  R&D  satellite  program. 


0.0:  Validation  Process 


Figure  8:  Final  Validation  Process 
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Exercises:  Exercises  are  mainly  used  as  training  events,  but  they  have  a  secondary 
mission  of  testing  the  ground  system  and  operational  concept  under  realistic  loading 
conditions.  Also  Exercises  are  often  executed  as  needed  throughout  the  validation 
process.  Because  of  this,  our  SMEs  concluded  that  they  did  not  need  to  be  individually 
called  out  in  our  process. 

Week  in  the  Eife  Tests:  Week  in  the  Eife  Tests  (WETEs)  are  performed  during  the 
readiness  phase  of  a  mission.  The  WITE  is  used  to  ensure  that  the  system  can  handle  the 
loading  of  nominal  operations.  Our  SMEs  feedback  was  that  typically  only  one  WITE  is 
performed  for  an  R&D  satellite  mission,  so  we  deleted  the  second  WITE  from  our  CVM. 

Rehearsals:  Eike  Exercises,  Rehearsals  are  mainly  used  as  training  events,  but  they 
have  a  secondary  mission  of  testing  the  ground  system  and  operational  concept  under 
realistic  loading  conditions.  Also  Rehearsals  are  often  executed  as  needed  throughout  the 
validation  process.  Because  of  this,  our  SMEs  concluded  that  they  did  not  need  to  be 
individually  called  out  in  our  process. 

Day  in  the  Eife  Tests:  Day  in  the  Eife  Test  (DITE)  is  the  only  step  in  the  validation 
process  that  was  added  based  on  SME  feedback.  The  DITE  exercises  the  system  based 
on  a  normal  day’s  activities  (not  a  Eaunch  and  Early  Orbit  (EEO)  activities.)  The  main 
goal  is  to  identify  any  deficiencies  with  the  ground  system  that  would  prevent  normal 
operations.  A  secondary  goal  is  to  examine  the  routine  operational  usability  of  the 
system  at  a  point  where  there  is  still  some  ability  to  make  modifications  if  a  more 
efficient,  or  better  process  can  be  established.  Routine  procedures  should  be  run.  Post 
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pass  processing  of  data  should  be  completed.  Everything  should  work  as  expected  on- 


orbit  or  changes  to  the  Concept  of  Operations  (CONOPS)  or  the  ground  system  need  to 
be  made.  The  focus  is  mainly  on  the  ground  system’s  ability  to  perform  the  procedures 
and  CONOPS. 

4.3  Costs  Associated  with  Executing  Each  Validation  Step 

Operational  Acceptance  Testing:  If  Operational  Acceptance  Testing  (OAT)  is 
executed  as  a  separate  step  it  typically  costs  approximately  $12,600.  OAT  is  conducted 
by  the  operators  with  little  to  no  outside  support.  This  allows  the  most  realistic 
assessment  of  operational  objectives.  Table  3  is  a  cost  breakdown  of  OAT. 


Table  3:  OAT  Cost  Analysis 


OAT 

Contract 

Hours 

GS  Development  Contractor 

0 

$0.00 

Operations  Contractor 

168 

$12,600.00 

Test  Asset 

0 

$0.00 

Satellite  Development  Contractor 

0 

$0.00 

$12,600.00 

Exercises:  The  number  of  exercises  executed  by  an  operations  team  is  based  on 
operator  experience  and  uniqueness  of  the  mission  objectives.  Table  4  shows  the  typical 
cost  of  one  exercise,  which  is  $25,300.  The  operators  conduct  them  and  the  ground 
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system  development  eontraetor  has  one  person  on  standby  to  resolve  ground  system  (GS) 


issues.  A  typical  program  will  execute  between  one  and  three  exercises. 


Table  4:  Exercise  Cost  Analysis 


Exercise 

Contract 

Hours 

Dollars 

GS  Development  Contractor 

40 

$4,000.00 

Operations  Contractor 

284 

$21,300.00 

Test  Asset 

0 

$0.00 

Satellite  Development  Contractor 

0 

$0.00 

$25,300.00 

Data  Flow  Testing:  Data  Flow  Tests  (DFT)  are  usually  executed  by  connecting  the 
satellite  and  ground  system  through  a  mobile  communication  system  and  a  T-1  line.  No 
Radio  Frequency  (RF)  functionality  is  tested  during  this  step.  The  cost  of  a  DFT  is 
$123,150.  The  satellite  and  ground  system  contractors  usually  conduct  the  DFT.  The 
operators  observe  this  test,  but  do  not  actively  participate.  Table  5  is  the  cost  breakout 
for  DFT. 


Table  5:  DFT  Cost  Analysis 


DFT 

Contract 

Hours 

Dollars 

GS  Development  Contractor 

702 

$70,200.00 

Operations  Contractor 

252 

$18,900.00 

Test  Asset 

0 

$0.00 

Satellite  Development  Contractor 

454 

$34,050.00 

$123,150.00 
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Factory  Compatibility  Test:  Factory  Compatibility  Test  (FCT)  can  be  executed  using 


one  of  two  sets  of  equipment  proyided  by  the  organization  that  proyides  test  assets  to  the 
Space  Community.  The  first  choice  is  the  Transportable  Space  Test  and  Eyaluation 
Resource  (TSTR)  yan.  TSTR  is  an  exact  replica  of  an  Air  Force  Satellite  Control 
Network  (AFSCN)  Remote  Tracking  Station  (ARTS).  The  second  choice,  used  if  TSTR 
is  not  ayailable,  is  S-Band  Transportable  Ground  System-T  (STGS-T).  It  is  used  in 
conjunction  with  manual  calculations  to  ensure  the  accuracy  of  the  Inter-Range  Operating 
Number  (IRON)  Database.  The  operators  and  satellite  contractor  execute  the  FCT  with 
support  from  the  ground  system  contractor.  The  cost  profiles  for  an  FCT  executed  with 
TSTR  and  an  FCT  execute  with  STGS-T  are  below  in  Table  6.  The  typical  cost  of  an 
FCT  is  between  $333,600  and  $393,600. 


Table  6:  FCT  Cost  Analysis 


FCT  with  TSTR 

FCT  With  STGS-T 

Contract 

Hours 

Dollars 

Contract 

Hours 

Dollars 

GS  Development  Contractor 

296 

$29,600.00 

GS  Development  Contractor 

296 

$29,600.00 

Operations  Contractor 

412 

$30,900.00 

Operations  Contractor 

412 

$30,900.00 

Test  Asset  Site  Survey 

$60,000.00 

Test  Asset  Site  Survey 

$60,000.00 

Test  Asset 

$220,000.00 

Test  Asset 

$160,000.00 

Satellite  Development  Contractor 

708 

$53,100.00 

Satellite  Development  Contractor 

708 

$53,100.00 

$393,600.00 

$333,600.00 

Week  in  the  Life  Tests:  A  WITL  includes  participation  from  the  satellite  contractor, 
the  ground  system  contractor  and  the  operator.  The  typical  cost  of  a  WITL  shown  in 
Table  7  is  $58,000. 
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Table  7:  WITL  Cost  Analysis 


WITL 

Contract 

Hours 

Dollars 

GS  Development  Contractor 

40 

$4,000.00 

Operations  Contractor 

360 

$27,000.00 

Test  Asset 

0 

$0.00 

Satellite  Development  Contractor 

360 

$27,000.00 

$58,000.00 

Command  Validation:  The  operators,  with  minimal  support  from  the  ground  system 
contractor,  execute  Command  Validation  (CV).  The  satellite  contractor  typically 
produces  the  “truth  data”  for  the  event,  or  completes  the  analysis  to  ensure  compatibility 
of  all  commands.  The  typical  cost  of  CV  shown  in  Table  8  is  $19,900. 


Table  8:  CV  Cost  Analysis 


CV 

Contract 

Hours 

Dollars 

GS  Development  Contractor 

40 

$4,000.00 

Operations  Contractor 

106 

$7,950.00 

Test  Asset 

0 

$0.00 

Satellite  Development  Contractor 

106 

$7,950.00 

$19,900.00 

Telemetry  Validation:  Telemetry  Validation  (TV)  is  executed  by  the  operators, 
sometimes  in  conjunction  with  CV  and  with  minimal  support  from  the  ground  system 
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contractor.  The  satellite  eontraetor  typieally  produees  the  “truth  data”  for  the  event,  or 


completes  the  analysis  to  ensure  compatibility  of  all  telemetry.  The  typical  cost  of  TV 
shown  in  Table  9  is  $19,900. 


Table  9:  TV  Cost  Analysis 


TV 

Contract 

Hours 

Dollars 

GS  Development  Contractor 

40 

$4,000.00 

Operations  Contractor 

106 

$7,950.00 

Test  Asset 

0 

$0.00 

Satellite  Development  Contractor 

106 

$7,950.00 

$19,900.00 

Rehearsals:  The  typical  cost  of  one  rehearsal,  shown  in  Table  10  is  $141,300. 
Rehearsals  are  executed  with  participation  from  the  entire  Mission  Control  Force  (MCF). 
The  MCF  includes  the  operator,  the  members  of  the  satellite  contract  that  will  be  present 
during  the  LEO  phase  of  the  mission  and  the  payload  specialists.  The  ground  system 
development  contractor  has  one  person  on  standby  to  resolve  ground  system  issues.  A 
typical  program  will  execute  three  rehearsals. 
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Table  10:  Rehearsal  Cost  Analysis 


Rehearsals 

Contract 

Hours 

Dollars 

GS  Development  Contractor 

120 

$12,000.00 

Operations  Contractor 

524 

$39,300.00 

Test  Asset 

0 

$0.00 

Satellite  Development  Contractor 

1200 

$90,000.00 

$141,300.00 

Day  in  the  Life  Test:  A  DITL  includes  participation  from  the  satellite  contractor,  the 
ground  system  contractor  and  the  operator.  The  typical  cost  of  a  DITL  is  shown  in  Table 
1 1 .  The  typical  cost  is  $  1 1 ,600. 


Table  11:  DITL  Cost  Analysis 


DITL 

Contract 

Hours 

Dollars 

GS  Development  Contractor 

8 

$800.00 

Operations  Contractor 

72 

$5,400.00 

Test  Asset 

0 

$0.00 

Satellite  Development  Contractor 

72 

$5,400.00 

$11,600.00 

Launch  Based  Compatibility  Test:  An  LBCT  can  be  executed  in  seyeral  different 
ways  depending  on  the  launch  site,  launch  configuration,  budget,  and  schedule  of  the 
R&D  program.  The  recommended  way  to  execute  an  LBCT  is  to  use  ARTS  and  execute 
the  LBCT  once  the  satellite  has  been  integrated  with  the  launch  yehicle.  Howeyer,  this  is 
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only  feasible  if  the  R&D  satellite  is  launching  from  a  location  with  ARTS,  such  as  the 
Eastern  or  Western  Range.  Another  advantage  of  this  is  that  use  of  ARTS  does  not 
require  the  additional  costs  of  a  test  asset.  It  is  scheduled  like  a  normal  satellite  support. 
If  the  satellite  is  not  launching  from  the  Eastern  or  Western  Range,  TSTR  or  STGS-T  can 
be  shipped  to  the  launch  site  to  perform  the  LBCT.  This  provides  the  same  amount  of 
risk  mitigation  as  ARTS,  but  shipment  of  TSTR  or  STGS-T  is  expensive,  especially  if  the 
launch  site  is  secluded,  like  Kwajalein  Atoll.  Because  of  this,  some  program  offices  may 
elect  to  perform  their  final  compatibility  validation  test  at  the  factory  before  shipment  of 
the  satellite.  They  will  perform  limited  RE  testing  with  the  satellite  at  the  launch  site  to 
ensure  nothing  was  damaged  during  the  shipment.  This  is  referred  to  as  a  Validation 
Eactory  Compatibility  Test  (VECT).  It  is  performed  in  the  place  of  an  LBCT  in  certain 
situations.  Because  of  the  variety  of  ways  this  test  can  be  executed  the  range  of  costs  is 
large.  The  least  expensive  LBCT  alternative  is  the  ARTS  option  at  $133,900.  The  most 
expensive  option  is  LBCT  at  Launch  Site  with  TSTR  at  $415,500.  The  remaining 
alternatives  fall  in  between.  The  costs  of  LBCT  are  shown  in  Table  12  below. 
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Table  12:  LBCT  Cost  Analysis 


VFCT  at  Factory  with  TSTR 

VFCT at  Factory  with  STGS-T 

Contract 

Hours 

Dollars 

Contract 

Hours 

Dollars 

GS  Development  Contractor 

296 

$29,600.00 

GS  Development  Contractor 

280 

$28,000.00 

Operations  Contractor 

412 

$30,900.00 

Operations  Contractor 

412 

$30,900.00 

Test  Asset 

$220,000.00 

Test  Asset 

0 

$160,000.00 

Satellite  Development  Contractor 

708 

$53,100.00 

Test  Asset  Site  Survey 

$60,000.00 

$333,600.00 

Satellite  Development  Contractor 

708 

$53,100.00 

$304,000.00 

LBCT  at  ARTS 

LBCT  at  La  unch  Site  with  TSTR 

LBCT  at  Launch  Site  with  STGS-T 

Contract 

Hours 

Dollars 

Contract 

Hours 

Dollars 

Contract 

Hours 

Dollars 

GS  Development  Contractor 

280 

$28,000.00 

GS  Development  Contractor 

296 

$29,600.00 

GS  Development  Contractor 

280 

$28,000.00 

Operations  Contractor 

412 

$30,900.00 

Operations  Contractor 

412 

$30,900.00 

Operations  Contractor 

412 

$30,900.00 

Test  Asset 

0 

$0.00 

Test  Asset  Ops 

$220,000.00 

TestAsset  Ops 

0 

$160,000.00 

Satellite  Development  Contractor 

1000 

$75,000.00 

Test  Asset  Site  Survey 

$60,000.00 

Test  Asset  Site  Survey 

$60,000.00 

$133,900.00 

Satellite  Development  Contractor 

1000 

$75,000.00 

Satellite  Development  Contractor 

1000 

$75,000.00 

$415,500.00 

$353,900.00 

All  of  the  steps  in  our  final  CVM  are  possible  steps  that  can  be  selected  when 
defining  a  validation  strategy.  Table  13  is  an  example  compatibility  validation  strategy 
illustrating  possible  total  costs  of  the  validation  effort.  In  the  next  section  we  will 
evaluate  the  risks  our  SMEs  defined  that  the  CVM  is  designed  to  mitigate. 


Table  13:  Example  Validation  Steps  and  Associated  Costs 


Test 

Option 

Costs 

Operational  Acceptance  Testing 

Operational  Acceptance  Testing 

$12,600.00 

Data  Flow  Testing 

Data  Flow  Testing 

$123,150.00 

Exercises 

Two  Exercises 

$50,600.00 

Rehearsals 

Three  Rehearsals 

$423,900.00 

Factory  Compatibility  Testing 

TSTR 

$393,600.00 

Week  in  the  Life  Testing 

Week  in  the  Life  Testing 

$58,000.00 

Telemetry  Validation 

Telemetry  Validation 

$19,900.00 

Command  Validation 

Command  Validation 

$19,900.00 

Launch  Based  Compatibility  Testing 

TSTR  at  Launch  Site 

$415,500.00 

Day  in  the  Life  Test 

Day  in  the  Life  Test 

$11,600.00 

$1,528,750.00 
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4.4  Strategy  Based  Risk  Model 

During  the  Delphi  Method,  we  asked  our  SMEs  to  list  all  of  the  risks  associated  with 
the  compatibility  between  a  satellite  and  its  ground  system.  These  risks  make  up  the 
Strategy  Based  Risk  Model  (SRM)  for  ground  system  and  satellite  validation  testing.  In 
order  to  develop  the  SRM  we  asked  them  to  define  the  risk  attributes  and  variables.  Our 
SMEs  identified  18  risks.  Table  14  below  lists  these  18  risks,  a  description  of  each  risk 
and  the  steps  in  the  validation  process  that  mitigate  the  risk.  Also  included  are  the 
probabilities  of  each  risk  being  realized  before  executing  the  validation  process,  the 
probability  of  each  risk  being  realized  after  executing  the  validation  process,  the  severity 
of  the  impact  of  realizing  the  risk,  and  the  costs  associated  with  that  impact.  We  have 
also  included  these  risks  in  the  Space  Development  and  Test  Wing  (SDTW)  risk  matrix 
discussed  in  Chapter  3.  This  matrix  will  allow  program  offices  to  easily  assess  whether 
this  risk  will  be  acceptable  to  SDTW  leadership.  Table  15  displays  the  risk  profile  before 
executing  the  validation  process.  The  “B”  following  each  risk  number  indicates  that  the 
risk  probability  shown  is  before  the  validation  process  was  completed.  Table  16  displays 
the  risk  profile  after  the  validation  process  is  completed  assuming  all  risks  are  mitigated. 
The  “A”  following  each  risk  number  indicates  the  risk  probability  shown  is  after  the 
validation  steps  are  completed. 
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Table  14:  Final  Risk  Table 


Risk 

Description 

Probability  before 
validation  step: 

Probability  after 
validation  step: 

Impact  Severity 

Impact  Costs 

Steps 

1 

If  SC  and  GSare  not  RF  Compatible  then  GS 
cannot  communicate  with  SC  and  mission 

is  lost 

41-60% 

0-10% 

Critical 

$100,000,000.00 

FCTand  LBCT 

2 

If  ARTS  Iron  Database  is  incorrect  then  GS 

cannot  communicate  with  SC  and  mission 

is  lost 

41-60% 

0-10% 

Critical 

$100,000,000.00 

FCT,  LBCT  and  CV 

3 

If  cmds  from  ground  system  do  not  execute 
properly  on  SC  then  new  command 
database  will  be  required  before 
commands  can  be  used 

41-60% 

0-10% 

Serious 

$1,000,000.00 

CV 

4 

If  there  are  telemetry  incompatibilities 
and  errors  between  the  SC  and  GS  then 
telemetry  may  be  reported  incorrectly 

41-60% 

0-10% 

Serious 

$1,000,000.00 

TV 

5 

Ifthere  are  telemetry  is  displayed 
incorrectlyon  GS  then  telemetry  may  be 
reported  incorrectly;  could  cause  operator 
to  incorrectly  assess  SOH  of  SC  and  take 
improper  measures 

41-60% 

0-10% 

Serious 

$1,000,000.00 

TV,  FCTand  DFT 

6 

If  Ground  system  software  does  not 
construct  and  release  spacecraft 
command  correctly  then  software  will  have 
to  be  modified  before  commands  can  be 

sent  to  the  SC 

11-40% 

0-10% 

Critical 

$100,000,000.00 

CV,  FCTand  DFT 

7 

If  Ground  system  is  unable  correctly  post¬ 
pass  process  payload/mission  data  then 
customerwill  notgetthere  data 

41-60% 

0-10% 

Moderate 

$200,000.00 

WITL,  FCT 

8 

Ifthere  are  data  latency  impacts  based  GS 
processessingtime  then  Customerwill  be 
delayed  In  receiving  data 

41-60% 

11-40% 

Serious 

$1,000,000.00 

OAT,  Exercises,  Rehearsals,  DFT, 

FCTand  LBCT 

9 

If  vehicle  manufacture  tries  something  new 
with  command,  format  then  it  could  cause 
compatibility  problems  between  ground 
system  and  spacecraft. 

61-90% 

11-40% 

Serious 

$150,000.00 

DFT,  FCT,  LBCT,  CV  and  TV 

10 

Ifthere  is  insufficient  documentation  on 

the  SC  for  the  GS  manufacturer  then  GS 
manufacture  incorrectlycode  software  and 
it  will  not  be  compatible  with  SC 

61-90% 

11-40% 

Minor 

$24,000.00 

OAT,  DFT,  FCTand  LBCT 

11 

Ifthere  is  insufficient  documentation  on 

the  GS  for  the  SC  manufacturer  then  SC 
manufacture  will  build  capability  that 
ground  system  can't  handle,  ground 
system 

41-60% 

0-10% 

Minor 

$24,000.00 

OAT,  DFT 

12 

If  SC  is  not  mature  then  GS  development 
will  have  to  change  tocontinue  to  be 
compatible  with  the  SC 

11-40% 

0-10% 

Serious 

$150,000.00 

CVandTV 

13 

If  ground  system  development  is  immature 
then  ground  system  will  have  last  minutes 
changes  that  will  increase  costs 

11-40% 

0-10% 

Moderate 

$65,000.00 

OAT,  FCT,  LBCTand  DFT 

14 

If  the  SC  manufacturer  does  not  provide 
telemetry  truth  data  then  telemetry  may 
not  be  processed  correctly  resulting  in  SW 
fixes 

41-60% 

11-40% 

Minor 

$65,000.00 

DFT,  TV,  FCTand  LBCT 

15 

If  the  SC  manufacturer  does  not  provide 
vehicle  command  samples  then  commands 
may  not  be  compatible  with  SC 

41-60% 

11-40% 

Minor 

$6,000.00 

DFTand  CV 

16 

Ifthe  system  fails  to  support  all 
operational  requirements  ofthe  satellite 
then  large  changes  will  have  to  be  made  to 
GSto  fix 

11-40% 

0-10% 

Moderate 

$500,000.00 

OAT,  Exercises,  Rehearsals,  WITL, 

DITL 

17 

If  GS  loses  or  corrupts  SC  data  then 
Satellite  commanding  and  telemetry  will 
be  erratic 

41-60% 

0-10% 

Minor 

$100,000.00 

OAT,  CVandTV 

18 

If  Post  Pas  processing  does  not  format 
products  correctly  then  delivered  products 
will  be  improperlyformatted  (i.e.  tasking 
files) 

0-10% 

0-10% 

Minor 

$100,000.00 

TV,  WITL,  exercises  and  rehearsal 
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Table  15:  Final  Risk  Matrix  Before  Validation 


Table  16:  Final  Risk  Matrix  After  Validation 


As  we  can  see,  from  the  Final  Risk  Table  and  the  Risk  Matrices  above,  Risk  #18,  “If 
Post  Pass  processing  does  not  format  products  correctly  then  delivered  products  will  be 
improperly  formatted  (i.e.  tasking  files),”  already  resides  in  the  lowest  risks  probability 
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range:  0-10%,  and  therefore  cannot  be  mitigated  into  a  new  probability  range.  Risk  #18 
also  has  a  Minor  Severity. 

4.5  How  to  Apply  the  Simulation 

The  purpose  of  our  thesis  was  to  create  a  simulation  that  can  be  used  by  program 
offices  to  help  develop  a  validation  strategy  that  provides  the  best  balance  of  risk 
mitigation  and  costs.  The  simulation  we  created  is  easy  for  the  program  offices  to  use. 
We  created  our  simulation  by  using  the  empirical  decision  aided  method  for  selecting  a 
process  to  validate  an  R&D  satellite  and  its  ground  system,  and  the  SRM  for  ascertaining 
risk,  probability  of  impact,  and  severity  of  impact. 

The  program  office  can  use  the  results  of  our  simulation  to  select  a  preferred 
validation  strategy  and  defend  it  to  senior  leadership.  It  can  also  be  used  to  illustrate  the 
fact  that  various  steps  in  the  validation  process  can  be  skipped  with  little  impact  to  the 
mission.  For  a  program  office  to  use  our  simulation,  they  will  follow  a  decision  analysis 
plan.  This  section  will  demonstrate  how  this  works. 

First  the  program  office  will  need  to  identify  which  steps  in  the  validation  process  they 
would  like  to  execute.  The  validation  steps  that  the  program  office  wishes  to  perform 
will  be  selected  from  drop  down  menus  in  the  “Summary”  sheet  of  our  simulation.  Figure 
9  below.  As  stated  in  Section  4.3,  some  steps  will  be  executed  differently  depending  on 
the  circumstances  of  the  program.  For  example  an  FCT  can  be  executed  with  TSTR  if  it 
is  available,  but  if  it  is  not  available  the  program  office  may  select  to  execute  an  FCT 
using  STGS-T. 
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Test 

Total  Life  Cycle  Costs  with 

Option  Realized  Risk 

Operational  Acceptance  Testing 

Operational  Acceptance  Testing  Before  Validation 

After  Validation 

Data  Flow  Testing 

Data  Flow  Testing  $68,059,333.33 

$1,033,100.00 

Exercises 

Two  Exercises 

Rehearsals 

Three  Rehearsals 

Factory  Compatibility  Testing 

TSTR 

Week  in  the  Life  Testing 

Weekinthe  Life  Testing 

Telemetry  Validation 

Telemetry  Validation 

Command  Validation 

Command  Validation 

Launch  Based  Compatibility  Testing 

TSTRat  Launch  Site 

Dayinthe  Life  Test 

None 

Test 

Costs 

Operational  Acceptance  Testing 

$12,600.00 

Data  Flow  Testing 

$123,150.00 

Exercises 

$50,600.00 

Rehearsals 

$423,900.00 

Factory  Compatibility  Testing 

$393,600.00 

Week  in  the  Life  Testing 

$58,000.00 

Telemetry  Validation 

$19,900.00 

Command  Validation 

$19,900.00 

Launch  Based  Compatibility  Testing 

$415,500.00 

Dayinthe  Life  Test 

$0.00 

Total  Costsof  Validation 

$1,517,150.00 

Risks 

Short  Title 

1 

RF  Compatibility 

2 

RTS/SC  Incompatibility 

Negligible 

Minor 

Moderate 

Serious 

3 

GS/SC  Command  Incompatibility 

0-10% 

ll^  17A,18A 

7A,  13A,16A 

3A,4A,  5A,12A 

4 

GS/SCTelemetry  Incompatibility 

11-40% 

lOA,  14A,  15A 

8A,9A 

5 

TLM  Displays 

41-60% 

6 

SC  Commands 

61-90% 

7 

Post  Pass  Processing 

91-100% 

8 

Data  Latency 

9 

New  Command  Format 

10 

SC  Documentation 

11 

GS  Documentation 

12 

Maturity  of  SC 

13 

Maturity  of  GS 

14 

SCTruth  Data 

15 

SC  Command  Samples 

16 

Ops  requirements 

17 

Lost  Data 

18 

PPP  Formatting 

Figure  9:  "Summary"  Sheet  of  Simulation 


As  we  can  see  in  Figure  9  above,  a  step  in  the  validation  process  is  executed  using  a  drop 
down  menu  under  the  “Option”  column.  From  this  selection,  the  associated  costs  are 
populated  and  the  total  costs  for  a  validation  strategy  is  calculated  using  the  equation 
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presented  in  the  methodology.  The  SDTW  risk  matrix  is  also  populated  illustrating  what 
risks  have  been  mitigated  by  executing  the  selected  validation  strategy.  From  this,  the 
program  office  will  determine  their  risk  level  and  decide  if  it  is  an  acceptable  level  of  risk 
for  their  program.  Every  program  is  different,  but  most  tend  to  accept  risks  that  are  in  the 
green  and  the  yellow  categories  at  the  lowest  possible  probability  of  impact.  The 
program  office  can  reference  Table  14  to  note  what  other  steps  in  the  validation  process 
can  be  executed  to  mitigate  the  indentified  risks. 

Once  the  validation  steps  have  been  identified  and  the  simulation  is  set  up,  it  can  be 
run  one  time  by  pressing  “Control+M”  or  several  times  using  a  macro.  In  order  to  gain 
statistical  confidence,  we  ran  the  simulation  1,000  times  in  every  trial.  When  the 
simulation  is  running,  it  decides  whether  a  risk  is  an  occurrence  or  nonoccurrence  by 
selecting  a  random  number  for  each  risk.  Figure  10  is  a  screen  shot  of  the  “After”  sheet 
displaying  whether  each  individual  risk  was  an  occurrence  or  a  nonoccurrence.  If  there  is 
an  impact  cost  displayed  in  the  cell,  then  the  risk  was  an  occurrence,  if  the  cell  displays 
$0.00  then  the  risk  was  a  nonoccurrence.  For  risks  that  result  in  loss  of  mission,  the 
impact  costs  are  $100,000,000.  This  the  representative  cost  of  re-accomplishing  the 
mission.  This  number  can  be  changed  within  the  simulation  to  represent  the  actual  cost 
of  the  mission.  Although  only  8  risks  are  shown  in  Figure  10  below,  all  18  risks  are 
considered. 
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Riskl 

Risk  2 

Risk  3 

Risk  4 

Risks 

Risk  6 

Risk? 

Risks 

Run  1 

$100,000,000.00 

$0.00 

$0.00 

$1,000,000.00 

$1,000,000.00 

$0.00 

$0.00 

$0.00 

Run  2 

$0.00 

$100,000,000.00 

$1,000,000.00 

$0.00 

$0.00 

$0.00 

$0.00 

$0.00 

Run  3 

$100,000,000.00 

$100,000,000.00 

$0.00 

$0.00 

$0.00 

$0.00 

$0.00 

$0.00 

Run  4 

$0.00 

$0.00 

$0.00 

$0.00 

$0.00 

$0.00 

$200,000.00 

$0.00 

Figure  10:  “After”  Sheet  of  Simulation 


From  this  the  Risk  Impact  Costs  and  Overall  Validation  Costs  are  calculated  using  the 
equations  presented  in  our  methodology.  The  Overall  Validation  Costs  are  displayed  on 
the  “Overall  Validation  Costs”  sheet  of  our  simulation,  shown  in  Figure  11.  The  average 
Overall  Validation  Costs  are  displayed  on  the  “Summary”  sheet  in  Figure  9. 


Overall  Validation  Costs  Before  =  Sum  of  Validation  Costs  Before 

e  Sum  of  all  Impact  costs  Before 

$0.00 

$619,000.00 

Overall  Validation  Costs  After  =  Sum  of  Validation  Costs  After 

e  Sum  of  all  Impact  After 

$1,517,150.00 

$6,000.00 

Overall  Validation  Costs  Before  with  Realized  Risk  Before  Validation 

Overall  Validation  Costs  Afterwith  Realized  Risk  AfterValidation 

$102,274,000.00 

$1,517,150.00 

$101,904,000.00 

$1,582,150.00 

$200,930,000.00 

$1,797,150.00 

$619,000.00 

$1,523,150.00 

Figure  11:  "Overall  Validation  Costs"  Sheet  of  Simulation 


This  simulation  can  be  executed  as  many  times  as  the  program  office  desires  to 
compare  different  validation  strategies.  These  strategies  are  compared  in  order  select  the 
desired  strategy  for  an  individual  program  based  on  cost  and  risk  aversion.  Once  they 
have  completed  this  exercise,  the  program  office  can  go  to  their  leadership  and  list  out 
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their  program  risks  in  a  systematic  method,  how  they  wish  to  mitigate  them,  and  what  the 
total  validation  cost  could  be. 

4.6  Apply  Simulation  to  Sample  Program 

In  order  to  demonstrate  the  usability  and  accuracy  of  our  simulation  we  have  applied  it 
to  a  program  that  is  currently  performing  validation  steps  in  preparation  for  operations  at 
SDTW.  This  sample  program  currently  holds  a  launch  date  within  the  next  12  months 
and  will  be  operated  as  the  first  satellite  on  SDTW’s  multi-mission  ground  system. 

4.6.1  Define  the  Validation  Strategies 

This  program  has  a  limited  budget  for  validation  due  to  launch  vehicle  problems  that 
caused  significant  launch  slips  and  severe  costs  increases.  In  order  to  save  money,  the 
program  office  proposed  a  strategy  for  executing  certain  steps  in  the  validation  process 
and  omitting  others.  The  program  office  proposed  to  execute  all  validation  steps  with  the 
exception  of:  OAT,  WITL,  TV,  CV  and  DITL.  This  sample  program  is  currently  in  the 
middle  of  the  validation  process  and  thus  some  steps  have  already  been  performed.  The 
sample  program  has  executed  a  DFT,  FCT  and  the  first  of  two  planned  exercises.  The 
program  office  plans  to  execute  Exercises  and  Rehearsals  as  well  as  a  VFCT  using  TSTR 
at  the  factory  two  months  prior  to  launch.  Since  this  satellite  is  not  launching  from  either 
the  Eastern  or  Western  Range,  an  EBCT  with  ARTS  was  not  an  option.  The  program 
office  does  not  wish  to  send  TSTR  up  to  the  launch  site  (Kodiak,  AK)  due  to  the  costs 
associated  with  shipment.  The  satellite  development  contractor  will  conduct  testing  once 
the  satellite  has  reached  the  launch  site  to  ensure  that  it  wasn’t  damaged  during  shipment. 


63 


The  costs  associated  with  the  program  offices  proposed  validation  strategy  are  shown  in 
Table  17  below.  The  risk  matrix  yielded  by  this  proposed  validation  strategy  is  shown  in 
Table  18.  A  risk  in  the  final  risk  matrix  that  has  a  “B”  following  the  risk  number 
indicates  that  the  risk  was  not  mitigated  and  the  risk  probability  shown  is  the  same  as  the 
probability  before  the  validation  steps  were  completed.  As  we  can  see  in  this  table,  the 
risks  that  were  not  mitigated  are:  Risks  3,  4,  12  and  17. 


Table  17:  Summary  Program  Office  Proposed  Validation  Strategy 


Test 

Option 

Costs 

Operational  Acceptance  Testing 

None 

$0.00 

Data  Flow  Testing 

Data  Flow  Testing 

$123,150.00 

Exercises 

Two  Exercises 

$50,600.00 

Rehearsals 

Three  Rehearsals 

$423,900.00 

Factory  Compatibility  Testing 

TSTR 

$393,600.00 

Week  in  the  Life  Testing 

None 

$0.00 

Telemetry  Validation 

None 

$0.00 

Command  Validation 

None 

$0.00 

Launch  Based  Compatibility  Testing 

TSTR  at  Factory 

$333,600.00 

Day  in  the  Life  Test 

None 

$0.00 

Total  Costs  of  Validation 

$1,324,850.00 

Table  18:  Program  Office  Proposed  Validation  Strategy  Risk  Matrix 
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Because  this  sample  program  will  be  the  first  to  fly  on  the  multi-mission  ground 


system,  SMC  assigned  an  Independent  Readiness  Review  Team  (IRRT)  to  the  ground 
system.  IRRTs  are  risk  adverse  and  evaluate  all  programs  on  the  same  scale  regardless  of 
the  budget.  The  IRRT  has  recommended  a  strategy  for  the  validation  process.  The  IRRT 
feels  that  all  steps  in  the  validation  process  were  necessary  with  the  exception  of  DITL. 

In  addition,  the  IRRT  feels  that  it  was  necessary  to  perform  an  LBCT  at  the  launch  site. 
The  IRRT  insists  that  separate  CV  and  TV  are  conducted  as  well  as  an  OAT  and  WITL. 
The  costs  associated  with  IRRTs  proposed  validation  strategy  are  shown  in  Table  19 
below.  The  risk  matrix  yielded  by  this  proposed  validation  strategy  is  shown  in  Table  20. 
This  risk  matrix  shows  that  the  IRRT  proposed  validation  strategy  mitigates  all  identified 
risks. 


Table  19:  Summary  IRRT  Proposed  Validation  Strategy 


Test 

Option 

Costs 

Operational  Acceptance  Testing 

Operational  Acceptance  Testing 

$12,600.00 

Data  Flow  Testing 

Data  Flow  Testing 

$123,150.00 

Exercises 

Two  Exercises 

$50,600.00 

Rehearsals 

Three  Rehearsals 

$423,900.00 

Factory  Compatibility  Testing 

TSTR 

$393,600.00 

Week  in  the  Life  Testing 

Week  in  the  Life  Testing 

$58,000.00 

Telemetry  Validation 

Telemetry  Validation 

$19,900.00 

Command  Validation 

Command  Validation 

$19,900.00 

Launch  Based  Compatibility  Testing 

TSTR  at  Launch  Site 

$415,500.00 

Day  in  the  Life  Test 

None 

$0.00 

Total  Costs  of  Validation 

$1,517,150.00 
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Table  20:  IRRT  Proposed  Validation  Strategy  Risk  Matrix 


For  the  sample  program  we  have  also  proposed  an  author  recommended  validation 
strategy.  This  strategy  assumes  that  the  steps  in  the  validation  process  which  have 
already  been  executed  are  considered  “sunk  costs.” 

The  steps  that  have  already  been  successfully  completed  are  DFT,  FCT  and  the  first  of 
two  planned  exercises.  We  recommend  that  in  addition  to  the  steps  already  completed, 
Exercises  and  Rehearsals  be  selected  as  part  of  the  validation  process  because  they  have  a 
primary  objective  of  training  and  certification  of  the  MCF.  Since  every  R&D  satellite 
mission  is  unique,  each  R&D  mission  requires  training  and  certification  of  the  MCF. 
Because  of  this.  Exercises  and  Rehearsals  will  be  performed  and  should  be  used  as  steps 
in  the  validation  process  as  well  as  training  activities.  The  final  two  steps  in  the 
validation  process  that  we  feel  should  be  conducted  are  CV  and  TV.  These  should  be 
conducted  because  there  are  two  significant  risks.  Risk  #3  and  Risk  #4,  which  can  only 
be  mitigated  by  CV  and  TV  respectively.  Risk  #3  is:  If  commands  from  ground  system 
do  not  execute  properly  on  the  satellite  then  a  new  command  database  will  be  required 
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before  commands  can  be  used.  A  full  CV  will  ensure  that  every  command  executes  as 


designed  onboard  the  satellite.  Risk  #4  is:  If  there  are  telemetry  incompatibilities  and 
errors  between  the  satellite  and  ground  system  then  telemetry  may  be  reported 
incorrectly.  TV,  which  includes  the  validation  of  real-time  telemetry  as  well  as  post-pass 
processed  files  and  limits,  ensures  that  telemetry  is  being  processed  correctly  by  the 
ground  system  and  thus  communicated  correctly  to  the  operators.  The  costs  of  our 
proposed  validation  strategy  considering  “sunk  costs”  are  shown  in  Table  21  below.  The 
risk  matrix  yielded  by  this  proposed  validation  strategy  is  shown  in  Table  22.  This  risk 
matrix  shows  that  our  proposed  validation  strategy  mitigates  all  identified  risks. 


Table  21:  Summary  Author  Proposed  Validation  Strategy 


Test 

Option 

Costs 

Operational  Acceptance  Testing 

None 

$0.00 

Data  Flow  Testing 

Data  Flow  Testing 

$123,150.00 

Exercises 

Two  Exercises 

$50,600.00 

Rehearsals 

Three  Rehearsals 

$423,900.00 

Factory  Compatibility  Testing 

TSTR 

$393,600.00 

Week  in  the  Life  Testing 

None 

$0.00 

Telemetry  Validation 

Telemetry  Validation 

$19,900.00 

Command  Validation 

Command  Validation 

$19,900.00 

Launch  Based  Compatibility  Testing 

None 

$0.00 

Day  in  the  Life  Test 

None 

$0.00 

Total  Costs  of  Validation 

$1,031,050.00 
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Table  22:  Author  Proposed  Validation  Strategy  Risk  Matrix 


Each  of  these  proposed  validation  strategies  was  modeled  using  our  simulation.  Three 
trials  of  1,000  simulation  runs  each  were  completed.  Three  trials  were  preformed  to 
ensure  that  the  results  were  consistent  and  the  simulation  operated  as  designed.  The  three 
trials  of  1,000  runs  were  also  combined  into  one  trial  of  3,000  runs. 

4.6.2  Simulation  Results 

After  running  simulation  trials  on  all  three  validation  strategies,  we  created  histograms 
of  each  trial  so  that  they  could  be  compared.  We  also  calculated  the  mean,  standard 
deviation,  range,  and  median.  The  histograms  and  calculations  for  all  of  the  trials  can  be 
found  in  Appendix  I.  Figure  12  displays  a  comparison  of  all  three  validation  strategies 
based  on  the  3,000  run  trial. 
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Validation  Strategy  Comparison 
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Figure  12:  Validation  Strategy  Comparison 


As  we  can  see  from  the  histogram  in  Figure  12,  the  median  value  for  these  validation 
strategies  is  between  $1M  and  $3M.  Comparing  the  median  value  for  different  validation 
strategies  helps  assess  which  strategy  to  choose.  The  average  overall  validation  cost  for 
each  option  is  between  $17M  and  $20M.  Unfortunately,  the  average  overall  cost  is  not  as 
useful  for  decision  making  in  this  situation  because  of  the  large  range  of  overall 
validation  costs.  Figure  12  shows  that  there  is  not  only  a  very  large  range  within  the 
possible  overall  validation  costs,  but  also  there  is  a  large  area  where  the  overall  validation 
costs  did  not  “hit”  in  the  simulation  at  all.  This  is  an  interesting  phenomenon  that  is 
unique  to  unmanned  spaceflight.  Currently,  if  there  is  an  irresolvable  problem  with 
satellite  to  ground  system  compatibility  once  the  satellite  is  on-orbit,  the  mission  is  lost. 
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Three  of  the  risks  identified  by  our  SMEs  have  impacts  that  are  loss  of  mission.  These 
impact  costs  equal  the  cost  of  the  entire  mission,  approximately  $100M  dollars.  If  one  of 
these  risks  is  realized  then  the  overall  validation  cost  elevates  to  over  $100M.  If  two  of 
these  risks  are  realized  then  the  cost  rises  to  above  $200M  and  if  all  three  risks  are 
realized  then  the  cost  is  over  $300M.  Because  of  these  three  risks,  there  are  three  large 
bins  where  there  can  be  no  overall  validation  costs. 

We  will  present  our  findings  and  interpretations  based  on  the  sample  program.  Table 
23  compares  the  three  validation  strategies  examined  for  this  sample  program.  This  is 
done  by  comparing  the  cost  of  completing  the  validation  strategy,  the  average  overall 
validation  cost,  the  standard  deviation,  the  range  of  overall  validation  costs,  and  the 
median  overall  validation  cost  for  each  strategy. 


Table  23:  Comparison  of  Validation  Strategies 


Validation  Strategy  Costs 

Mean 

Range 

Standard  Deviation 

Median 

Option  1  All:  PM 

$1,324,850.00 

$18,970,647.13 

$301,129,400.00 

$40,253,166.62 

$2,924,850.00 

Option  2AII:  IRRT 

$1,517,150.00 

$17,130,988.89 

$202,071,000.00 

$38,064,722.27 

$1,817,150.00 

Option  3AII;  Author 

$1,031,050.00 

$18,179,638.00 

$201,321,000.00 

$40,144,600.32 

$1,405,050.00 

From  looking  at  the  median  we  can  conclude  that  both  Option  2  and  3  are  superior  to 
Option  1 .  Option  3  is  displayed  as  the  best  option  because  it  not  only  is  the  least 
expensive,  but  also  mitigates  all  of  the  risks.  As  we  compare  this  data  in  attempt  to 
identify  a  desired  validation  strategy,  it  is  important  to  note  that  there  is  no  right  answer. 
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Each  program  office  needs  to  use  the  simulation  provided  and  then  evaluate  their  results 
based  on  the  factors  that  are  most  important  to  them. 

4.7  Program  Manager’s  Input  on  Simulation 

In  order  to  demonstrate  the  accuracy  of  the  simulation  used  to  generate  distributions 
on  possible  costs  and  risk  outcomes  that  we  developed  during  this  thesis  we  provided  the 
simulation  to  two  program  managers.  We  asked  the  program  managers  to  run  the 
simulation  for  their  on-going  programs  and  answer  a  series  of  questions.  We  asked  them 
to  also  keep  in  mind  past  programs  when  answering  the  questions. 

We  sought  out  program  managers  that  were  separate  from  our  Delphi  group.  We 
believe  that  this  not  only  validated  the  simplicity  and  usability  of  our  simulation  program, 
but  also  helped  to  validate  the  CVM  and  SRM  developed  through  the  Delphi  Method. 
Finally,  we  wanted  to  solicit  input  on  the  risks  listed  in  Table  14.  We  wanted  to  ensure 
that  these  were  the  only  risks  that  an  R&D  satellite  program  would  encounter  during  the 
validation  of  its  ground  system  to  the  satellite  compatibility. 

To  validate  the  Delphi  Method,  we  asked  the  program  managers  questions  regarding 
the  risks  we  identified  in  Table  14.  We  first  asked  them  if  the  risks  listed  in  the  table 
provided  (Table  14)  encompassed  all  the  risks  on  their  current  satellite  program. 

Program  manager  #1,  stated  that  their  program  risks  and  the  program  risks  we  identified 
were  the  same.  They  are  currently  tracking  12  risks,  all  of  which  can  be  found  on  our 
table.  However,  some  of  the  wording  for  the  risks  is  slightly  different.  The  program 
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manager  for  program  #2  stated  they  combined  some  of  their  risks  but  they  were  found 
within  our  table. 

We  also  asked  both  program  managers  if  they  had  ever  worked  an  R&D  satellite 
program  that  tracked  a  risk  concerning  the  compatibility  of  the  satellite  and  its  ground 
system  not  listed  in  Table  14.  We  received  a  unanimous  no.  We  did  get  the  comment 
that  some  of  the  risks  we  listed  in  the  table  would  not  be  considered  a  high  enough  risk  to 
track. 

To  evaluate  the  simulation,  we  asked  the  program  managers  their  thoughts  on  the 
simulation.  Specifically,  what  were  the  results  using  the  simulation  on  their  program? 
Can  using  the  simulation  save  their  programs  cost,  or  schedule?  Finally,  how  could 
utilizing  the  simulation  help  their  program?  Program  manager  #1  stated  that  utilizing  the 
simulation  could  help  them  eliminate  some  validation  steps  to  save  schedule  and  cost 
while  adequately  mitigating  risks  to  the  program.  They  also  believe  that  if  they  held  a 
successful  FCT,  they  could  possibly  save  both  cost  and  schedule  associated  with 
executing  an  LBCT.  The  program  manager  for  program  #1  stated  that  they  lost  a  lot  of 
schedule  due  to  performing  an  LBCT  on  a  past  program  that  may  not  have  been 
necessary  due  to  the  programs  successful  FCT.  Overall,  they  thought  they  could 
eliminate  costly  testing  with  the  aid  of  the  simulation.  Finally,  program  manager  #1 
believes  that  this  simulation  could  help  advocate  for  a  more  comprehensive  set  of  tests 
that  would  cut  down  on  more  risks. 
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Program  manager  #2  felt  very  similarly  to  program  manager  #1 .  However,  it  should 
be  noted  that  program  manager  #2  frequently  discussed  how  with  an  unlimited  budget,  it 
is  always  best  to  complete  as  much  testing  as  possible.  Additionally,  they  made  a 
number  of  statements  indicating  that  any  process  (simulation)  would  need  proper  push 
from  leadership  to  be  effective.  With  these  comments  noted,  they  did  state  that  the 
simulation  could  help  defend  skipping  validation  steps,  which  would  save  time  and 
money.  Program  manager  #2  believed  that  the  simulation  helped  provide  evidence  to  the 
team  showing  which  steps  in  the  validation  process  are  absolutely  necessary. 

The  program  managers  were  allowed  to  make  any  addition  comments  to  questions  we 
asked  them.  Program  manger  #1  really  liked  the  simulation  program  and  was  excited  to 
use  it  in  the  future.  Program  manager  #2  was  slightly  concerned  that  the  simulation 
program  often  pointed  to  not  completing  LBCTs  and  saw  this  as  a  potential  problem. 

This  could  be  a  potential  problem  because  not  conducting  an  LBCT  is  a  very  political 
issue.  Not  completing  this  test  can  be  a  very  unpopular  decision  regardless  of  the  lack  of 
technical  risk. 
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V.  Conclusion  and  Recommendations 


5.1  Introduction 

When  we  started  this  thesis,  we  set  out  to  create  a  simulation  that  would  generate 
distributions  of  possible  risk  and  cost  outcomes.  This  simulation  would  in  turn  aid 
Research  and  Developmental  (R&D)  satellite  program  offices  in  analyzing  the  risks 
associated  with  the  compatibility  between  satellites  and  their  ground  systems.  We 
wanted  to  help  these  program  offices  decide  which  risks  need  to  be  mitigated  and  which 
risks  can  be  accepted  at  their  current  level.  We  also  wanted  to  help  them  decide  the  more 
efficient  way  to  mitigate  these  risks  through  the  validation  process.  Throughout  this 
chapter  we  will  be  discussing  the  conclusions  and  significance  of  our  research. 

5.2  Conclusions  of  Research 

This  thesis  examined  readiness  and  on-orbit  activities  of  R&D  satellite  programs  and 
attempted  to  accurately  define  the  process  for  validating  the  compatibility  between  a 
satellite  and  its  ground  system.  Using  historical  program  data  together  with  Subject 
Matter  Experts  (SMEs),  our  thesis  mirrors  the  method  outlined  by  Engel  &  Barad  in  A 
Methodology  for  Modeling  WT  Risks  and  Costs.  Our  thesis  identified  all  of  the  steps  in 
the  process  of  validating  the  compatibility  between  a  satellite  and  ground  system.  We 
created  a  simulation  that  can  assist  program  offices  in  determining  which  steps  in  the 
validation  process  they  should  execute,  and  which  they  can  skip  based  on  the  risks. 
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Through  the  Delphi  Method  we  concluded  that  the  final  Canonical  Verification 
Validation  and  Testing  (VVT)  Model  (CVM)  presented  in  section  4.2.2  is  complete  and 
correct  CVM.  We  also  concluded  that  the  risks  identified  by  our  SMEs  through  the 
Delphi  Method  are  a  complete  and  accurate  list  of  risks.  The  attributes  and  variables  for 
these  risks  were  identified  via  the  Strategy  Based  Risk  Model  (SRM).  This  process 
allowed  us  to  create  a  simulation  that  generates  distributions  of  risk  and  cost  outcomes. 

If  this  simulation  is  executed  a  large  number  of  times,  conclusions  can  be  drawn  about 
how  much  of  the  budget  should  be  saved  for  contingency  costs.  If  the  simulation  is 
executed  with  more  than  one  validation  strategy,  it  can  be  used  as  a  comparison  tool  for 
selecting  a  desired  validation  method. 

We  gave  the  technique  and  simulation  to  two  program  managers.  These  program 
managers  evaluated  the  technique  and  provided  insight  into  the  usefulness  of  the 
technique.  In  the  future  we  will  be  distributing  this  technique  and  simulation  to  program 
offices  to  support  tailoring  a  validation  plan  based  on  their  budget.  The  technique  and 
simulation  will  give  decision  makers  insight  into  the  expected  risks  and  costs  associated 
with  the  selected  validation  process  so  that  they  can  make  informed  decisions.  They  will 
be  able  to  understand  and  accept  risks  with  reasonable  probability  and  severity  of 
impacts,  and  ensure  that  risks  with  unreasonable  probability  and  severity  of  impacts  are 
mitigated  to  the  fullest  extent  possible. 
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5.3  Significance  of  Research  and  Recommendations  for  Action 

The  technique  and  simulation  developed  in  this  thesis  will  be  invaluable  to  R&D 
satellite  program  offices.  These  program  offices  often  deal  with  diminishing  budgets.. 
The  typical  budget  for  an  R&D  satellite  program  office  at  Space  Development  and  Test 
Wing  (SDTW)  is  between  $300K  and  $400M.  Some  operational  satellites  have  budgets 
of  more  than  $1B. 

R&D  satellites  often  do  not  have  a  dedicated  launch  vehicle.  They  often  share  a 
launch  vehicle  with  as  many  as  6  other  satellites.  Because  of  this,  they  must  be  flexible 
with  their  schedules.  If  the  R&D  satellite  is  not  the  primary  mission  on  a  launch  vehicle, 
they  can  have  little  influence  on  the  launch  date,  and  the  launch  can  occur  whether  the 
R&D  satellite  is  ready  or  not.  Therefore  it  is  important  for  R&D  satellites  to  be  flexible 
and  responsive.  These  budget  and  schedule  constraints  present  R&D  program  offices 
with  the  unique  challenge  of  deciding  which  program  risks  to  mitigate  and  which  to 
accept. 

Our  technique  and  simulation  allow  program  offices  to  make  this  assessment  by 
providing  them  with  the  program  risks,  the  steps  in  the  validation  process  that  mitigate 
these  risks,  and  the  impacts  of  these  risks.  The  simulation  allows  the  program  offices  to 
generate  distributions  of  possible  risk  and  cost  outcomes  based  on  their  chosen  validation 
strategy.  Program  offices  can  compare  several  validation  strategies  to  help  decide  which 
strategy  is  best  for  them.  Using  our  technique  these  program  offices  will  be  able  to 
provide  data  to  defend  decisions.  They  will  be  able  to  explain  why  certain  steps  in  the 
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validation  process  are  necessary  and  why  other  steps  in  the  validation  process  are  not. 
While  our  research  was  completed  using  R&D  satellite  programs;  we  feel  there  is  another 
initiative  currently  in  DoD  that  can  apply  our  technique  and  simulation  as  well. 

Operationally  Responsive  Space  (ORS)  is  the  vision  for  the  future  of  military  space. 

In  future  conflicts,  military  space  forces  will  likely  face  challenges  ranging  from 
defending  against  opposing  systems  to  dealing  with  rapidly  changing  technology  and 
support  needs.  “The  goal  of  ORS  is  to  provide  an  affordable  capability  to  promptly, 
accurately,  and  decisively  position  and  operate  national  and  military  assets  in  and  through 
space  and  near  space.”  [Doggrell,  2006]  Since  the  goals  of  ORS  are  to  be  affordable  and 
responsive,  we  feel  that  our  risk  management  and  decision  analysis  technique  is  an 
excellent  way  to  help  accomplish  their  goals.  Our  recommendation  for  action  is  that  all 
SDTW  and  ORS  satellite  program  offices  adopt  this  technique  and  associated  simulation 
for  on-going  and  future  missions. 

5.4  Recommendations  for  Future  Research 

The  assumptions  and  scope  of  our  thesis  was  limited  to  R&D  satellite  programs.  As  a 
result,  we  feel  there  is  room  for  future  research  on  the  subject.  We  also  feel  that  the 
technique  presented  by  Engel  and  Barad  is  applicable  to  any  situation  where  a  series  of 
steps  is  performed  to  mitigate  risk.  For  example,  we  only  looked  at  the  compatibility 
between  the  satellite  and  ground  system.  The  technique  could  also  be  applied  to  satellite 
environmental  testing  and  launch  vehicle  testing  as  well  as  any  number  of  subjects 
within  and  outside  the  space  industry.  First,  this  research  can  be  expanded  into  domains 
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such  as  ORS.  Another  area  for  future  work  is  to  expand  the  proeess  to  more  accurately 
represent  reality. 

Our  validation  proeess  is  simplified  and  does  not  represent  reality.  There  are  eertain 
things  that  should  be  added  to  our  process  in  order  to  be  more  realistic.  Our  process 
assumes  all  testing  will  be  suecessful.  If  the  steps  identified  in  this  proeess  are  not 
successful  further  testing  will  need  to  be  performed.  In  order  the  make  our  process  more 
realistic  further  testing  can  be  represented  with  feedback  loops  and  logic  boxes.  Logic 
boxes  ean  be  included  to  ask  whether  the  test  is  successful.  If  the  test  is  successful  the 
process  can  proceed  to  the  next  step.  If  the  test  is  not  suecessful  the  proeess  will  be 
repeated.  This  is  an  area  that  requires  further  research  and  can  be  expanded  on  in  future 
work  within  this  subjeet  area. 

Another  assumption  we  made  that  limited  our  thesis  was  that  a  validation  step  can 
either  be  fully  performed  or  not  performed  at  all.  Sinee  testing  ean  be  partially 
completed,  this  is  an  area  for  future  researeh.  We  also  assumed  that  each  step  in  the 
validation  process  mitigates  the  risk  the  same  amount.  Future  work  ean  examine  if  some 
validation  steps  mitigate  risks  more  than  others. 

When  applying  our  simulation  we  noted  that  there  is  not  only  a  large  range  within  the 
possible  overall  validation  costs,  but  also  there  are  is  a  large  area  where  the  overall 
validation  costs  did  not  “hit”  at  all.  As  discussed  in  our  analysis,  this  is  because  the  loss 
of  mission  impact,  that  has  a  significantly  higher  impact  cost.  This  is  unique  to 
spaeeflight.  Future  works  should  compare  different  validation  strategies  not  considering 
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the  risks  that  result  in  loss  of  mission  and/or  only  considering  the  risks  that  result  in  loss 
of  mission.  This  will  reduce  the  range  and  produce  meaningful  averages  rather  than 
averages  that  are  skewed  due  to  the  large  range  of  data. 

The  final  area  for  future  research  that  we  have  identified  is  concentrating  on  reducing 
the  severity  of  the  impact  of  the  risk.  Currently  the  validation  process  only  mitigates  the 
probability  of  the  risk  being  realized.  Since  both  probability  and  severity  of  impact  are 
risk  variables,  it  would  be  useful  to  examine  what  steps  can  be  accomplished  to  mitigate 
the  severity  of  the  risk  in  addition  to  the  probability.  All  of  the  research  can  be 
accomplished  by  expanding  on  the  work  completed  by  Engel  and  Barad  and  by  ourselves. 

5.5  Summary 

In  summary  we  were  able  to  develop  a  technique  by  using  the  Delphi  Method  to 
evaluate  a  validation  process  for  compatibility  between  a  satellite  and  its  grounds  system. 
Through  the  Delphi  Method  we  were  able  to  determine  what  risks  were  associated  with 
ground  system  and  satellite  compatibility.  We  were  able  to  provide  a  point  estimate  for 
the  costs  associated  with  each  validation  step  and  ascertain  costs  and  severity  of  each  risk 
identified.  Finally  through  the  Delphi  Method  we  were  able  to  determine  what  validation 
steps  mitigated  which  risks.  This  technique  was  used  to  create  a  simulation  that 
generates  distributions  of  outcomes  based  on  risk  and  cost. 

The  technique  and  simulation  will  be  given  to  program  offices.  This  will  help  them 
save  time  when  determining  what  risks  their  program  has  for  ground  system  and  satellite 
compatibility.  It  will  also  allow  them  to  determine  the  best  validation  process  based  on 
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their  program  risks  and  budget  and  will  allow  them  to  go  to  their  leadership  with  this 
process,  showing  risks  before  and  after  validation  and  the  cost  associated  with  the 
validation  process. 
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Appendix  A:  Initial  Validation  Process 


Operational  Acceptance  Testing:  Operational  Acceptance  Testing  (OAT)  is 
performed  by  the  organization  that  will  be  operating  the  satellite  via  the  ground  system. 
The  operators  have  two  main  objectives  when  performing  this  testing.  (1)  Validate  that 
the  design  is  operationally  suitable  and  (2)  Evaluate  the  ground  system  under 
operationally  realistic  conditions. 

Exercise  1:  Exercise  one  is  a  training  event,  which  is  typically  the  first  time  the 
operators  use  the  ground  system.  Exercises  are  primarily  used  as  training  events,  but 
they  have  a  secondary  mission  of  testing  the  ground  system  and  operational  concept 
under  realistic  loading  conditions.  The  objectives  of  exercise  one  are  usually  to:  (1) 
Eamiliarize  the  operations  team  with  the  ground  system,  (2)  Assure  the  ground  system 
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design  is  feasible  under  realistic  loading  conditions,  (3)  Test  of  Concept  of  Operations, 
and  (4)  Familiarize  operators  with  procedures  and  processes. 

Data  Flow  Testing  (DFT):  Data  flow  testing  is  usually  the  first  time  the  satellite  and 
ground  system  are  allowed  to  interact.  It  is  the  first  opportunity  to  validate  that  the 
ground  system  can  receive  and  process  satellite  telemetry  and  that  the  satellite  receives 
and  processes  commands  sent  from  the  ground  system.  Problems  are  identified  with  the 
compatibility  between  the  satellite  and  the  ground  system.  The  DFT  often  identifies 
problems  before  they  impact  the  mission  and  schedule. 

Rehearsal  1:  Rehearsal  1  is  the  first  chance  to  train  the  entire  Launch  and  Early  Orbit 
(LEO)  operational  team.  The  operational  team  is  made  up  of  the  contractors  that  operate 
the  satellite,  the  flight  directors,  the  satellite  operations  crew  commanders,  the  satellite 
(satellite)  manufacturer  technical  advisors,  the  payload  technical  advisors  and  members 
of  the  program  office  or  independent  technical  advisors.  Rehearsals  are  primarily  used  as 
training  events,  but  they  have  a  secondary  mission  of  testing  the  ground  system  and 
operational  concept  under  realistic  loading  conditions.  The  objective  of  the  rehearsals  is 
to  certify  the  launch  and  early  orbit  operational  team.  Specifically,  the  objectives  are:  (1) 
Testing  the  ground  system  design  under  realistic  loading  conditions,  (2)  Testing  of 
Concept  of  Operations  and  (3)  Eamiliarizing  the  operator  with  procedures  and  processes. 
However,  the  first  rehearsal  is  90%  training  and  only  10%  certification. 

Eactory  Compatibility  Test:  The  ECT  is  performed  by  connecting  the  satellite  to  the 
ground  system  through  an  AESCN  test  van  or  the  Transportable  Space  Test  and 
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Evaluation  Resource  (TSTR)  van.  TSTR  is  operated  by  the  organization  that  provides  test 
assets  to  the  Space  Community.  This  organization  also  has  a  variety  of  other  equipment 
that  can  be  used  to  perform  this  test,  in  an  event  that  the  TSTR  van  is  not  available.  The 
TSTR  van  allows  us  to  validate  the  AFSCN  configuration  (ARTS  Inter-Range  Operating 
Number  (IRON)  Databases).  This  is  also  a  great  opportunity  to  perform  extensive 
satellite  and  ground  system  compatibility  testing.  The  primary  objective  of  the  FCT  is  to 
validate  the  IRON  databases.  The  FCT  is  completed  by  ensure  that  telemetry  is  received 
by  the  ground  system  and  commands  are  received  by  the  satellite.  Many  program  offices 
add  objectives  to  this  test  in  order  to  maximize  the  testing  opportunity.  These  objectives 
include:  (1)  Validation  of  all  command  types,  (2)  Validation  that  the  flight  software  and 
ground  software  are  compatible,  and  (3)  Ensure  that  critical  telemetry  points  are  being 
properly  processed  and  displayed  by  the  ground  system. 

Exercise  2:  The  second  exercise  has  the  same  objectives  as  the  first  exercise  and 
typically  only  involves  the  operations  contractor.  Exercises  are  primarily  used  as  training 
events,  but  they  have  a  secondary  mission  of  testing  the  ground  system  and  operational 
concept  under  realistic  loading  conditions. 

Week  in  the  Fife  Tests  1&2:  The  Week  in  the  Fife  Tests  (WITF)  are  performed 
during  the  readiness  phase  of  a  mission.  The  WITF  is  used  to  ensure  that  the  system  can 
handle  the  loading  of  nominal  operations.  This  test  is  also  used  to  validate  operational 
procedures.  The  WITF  is  most  useful  if  it  can  be  performed  between  the  ground  system 
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and  the  satellite.  However,  often  times  the  satellite  and  ground  system  perform  WITL 
separately  due  to  budgetary  constraints. 

Command  Validation:  The  purpose  of  command  validation  (CV)  is  to  ensure  that  the 
end  product  (the  commands  executed  by  the  satellite)  matches  the  input  (the  tasking  file 
provided  by  the  customer  or  the  commands  built  by  the  operator).  The  specific 
objectives  of  command  validation  are:  (1)  Validate  the  customer  provided  tasking  files 
are  processed  properly  by  the  ground  system,  and  (2)  Validate  the  command  database  and 
validate  that  a  specific  command  accomplishes  the  expected  action  on  the  satellite. 

Telemetry  Validation:  Telemetry  originates  on  the  satellite,  is  transmitted  to  the 
ground,  and  is  processed  by  the  ground  system.  The  objective  of  Telemetry  Validation 
(TV)  is  to  ensure  that  the  end  product  (the  raw,  processed,  and  displayed  telemetry) 
agrees  with  the  data  being  produced  on  the  satellite,  as  interpreted  in  accordance  with  the 
telemetry  database  provided  by  the  contractor.  The  steps  to  ensure  the  telemetry  is  valid 
are  as  follows:  (1)  Validating  the  raw  telemetry  at  the  output,  (2)  Validating  the 
processed  telemetry  products  (EU  converted  files,  etc.),  (3)  Validating  the  displayed 
telemetry  (telemetry  screens)  is  properly  converted  by  the  ground  system  and  (4) 
Validating  that  of  red,  yellow,  and  green  limits  are  handled  correctly. 

Rehearsal  2:  For  ground  system  validation,  the  second  rehearsal  has  the  same 
objectives  as  the  first  rehearsal.  However,  the  second  rehearsal  is  usually50%  for 
training  and  50%  for  launch  certification.  If  a  satellite  simulator  or  a  computer  running 
the  current  version  of  the  flight  software  is  used  for  the  rehearsal,  then  they  are  even 
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more  useful  for  satellite  to  ground  system  compatibility  validation.  This  ensures  that  the 
satellite  flight  software  and  ground  system  software  are  indeed  compatible. 

Launch  Based  Compatibility  Test:  The  launch  based  compatibility  test  (LBCT)  is  the 
final  opportunity  for  the  ground  system  and  satellite  to  connect  prior  to  launch.  All  of  the 
objectives  accomplished  at  FCT  are  revalidated  during  the  LBCT.  In  addition  to  the 
main  objectives  from  FCT,  LBCT  also  ensures  that  the  satellite  transponder  was  not 
damaged  during  the  shipment  of  the  satellite  to  the  launch  site  and  that  any  previously 
encountered  compatibility  issues  have  been  resolved.  Since  LBCT  is  the  last  chance  to 
confirm  compatibility,  the  ground  system  and  satellite  baseline  are  frozen  after  a 
successful  test,  and  no  software  or  hardware  changes  are  allowed  until  after  the  satellite 
has  launch  and  is  in  a  safe  configuration.  Freezes  prevent  the  team  from  inadvertently 
changing  something  that  does  not  allow  the  ground  system  to  contact  the  satellite. 

Rehearsal  Three:  For  ground  system  validation,  the  third  rehearsal  has  the  same 
objectives  as  the  first  and  second  rehearsals.  Rehearsals  are  primarily  used  as  training 
events,  but  they  have  a  secondary  mission  of  testing  the  ground  system  and  operational 
concept  under  realistic  loading  conditions.  However,  the  third  rehearsal  is  used  10%  for 
training  and  90%  for  launch  certification. 
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Mission  Dress  Rehearsal  (MDR):  MDR  is  the  final  training/validation  event  that 
occurs  before  launch,  usually  less  than  10  days  prior  to  launch.  Final  procedure 
acceptance  occurs  during  MDR.  It  is  also  the  final  validation  for  launch  critical 
commands.  This  is  completed  either  by  sending  this  commands  to  a  simulator  during 
MDR,  or  if  a  simulator  is  not  available,  by  bit  busting.  MDR  is  the  final  event  that 
Mission  Critical  Personnel  are  evaluated  and  certified.  It  is  also  the  final  validation  of 
system  interoperability. 
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Appendix  B:  Final  Validation 


0.0:  Validation  Process 


Operational  Acceptance  Testing:  Operational  Acceptance  Testing  (OAT)  is 
performed  by  the  organization  that  will  be  operating  the  satellite  via  the  ground  system. 
The  operators  have  two  main  objectives  when  performing  this  testing.  (1)  Validate  that 
the  design  is  operationally  suitable  and  (2)  Evaluate  the  ground  system  under 
operationally  realistic  conditions. 

Exercises:  Exercises  are  primarily  used  as  training  events,  but  they  have  a  secondary 
mission  of  testing  the  ground  system  and  operational  concept  under  realistic  loading 
conditions.  Exercises  are  executed  as  needed  throughout  the  validation  process.  Because 
of  this,  our  SMEs  concluded  that  they  did  not  need  to  be  individually  called  out  in  our 
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process.  The  number  of  exercises  executed  by  an  operations  team  is  based  on  operator 
experience  and  uniqueness  of  the  mission  objectives  and  not  on  their  secondary 
objectives  of  compatibility  validation. 

Data  Flow  Testing:  DFT  is  usually  the  first  time  the  satellite  and  ground  system 
interact.  It  is  the  first  opportunity  to  validate  that  the  ground  system  can  receive  and 
process  satellite  telemetry  and  that  the  satellite  can  receive  and  process  commands  sent 
from  the  ground  system.  DFT  is  usually  executed  by  connecting  the  satellite  and  ground 
system  through  a  mobile  communication  system  and  a  T-1  line.  No  RF  functionality  is 
tested  during  this  step. 

Factory  Compatibility  Test:  The  FCT  is  performed  by  connecting  the  satellite  to  the 
ground  system  through  an  AFSCN  test  van  or  the  Transportable  Space  Test  and 
Evaluation  Resource  (TSTR)  van.  TSTR  is  operated  by  the  organization  that  provides  test 
assets  to  the  Space  Community.  This  organization  also  has  a  variety  of  other  equipment 
that  can  be  used  to  perform  this  test,  in  an  event  that  the  TSTR  van  is  not  available.  The 
TSTR  van  allows  us  to  validate  the  AFSCN  configuration  (ARTS  IRON  Databases). 

This  is  also  a  great  opportunity  to  perform  extensive  satellite  and  ground  system 
compatibility  testing.  The  primary  objective  of  the  FCT  is  to  validate  the  IRON 
databases.  The  FCT  is  completed  by  ensure  that  telemetry  is  received  by  the  ground 
system  and  commands  are  received  by  the  satellite.  Many  program  offices  add  objectives 
to  this  test  in  order  to  maximize  the  testing  opportunity.  These  objectives  include:  (1) 
Validation  of  all  command  types,  (2)  Validation  that  the  flight  software  and  ground 
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software  are  compatible,  and  (3)  Ensure  that  critical  telemetry  points  are  being  properly 
processed  and  displayed  by  the  ground  system. 

Week  in  the  T3fe  Tests:  WETLs  are  performed  during  the  readiness  phase  of  a 
mission.  The  WITL  is  used  to  ensure  that  the  system  can  handle  the  loading  of  nominal 
operations.  Our  SMEs  feedback  was  that  typically  only  one  WITE  is  performed  for  an 
R&D  satellite  mission,  so  we  deleted  the  second  WITE  from  our  original  process.  A 
WIPE  is  most  successful  with  participation  from  the  satellite  contractor,  the  ground 
system  contractor  and  the  operator. 

Command  Validation:  The  purpose  of  command  validation  (CV)  is  to  ensure  that  the 
end  product  (the  commands  executed  by  the  satellite)  matches  the  input  (the  tasking  file 
provided  by  the  customer  or  the  commands  built  by  the  operator).  The  specific 
objectives  of  command  validation  are:  (1)  Validate  the  customer  provided  tasking  files 
are  processed  properly  by  the  ground  system,  and  (2)  Validate  the  command  database  and 
validate  that  a  specific  command  accomplishes  the  expected  action  on  the  satellite. 

Telemetry  Validation:  Telemetry  originates  on  the  satellite,  is  transmitted  to  the 
ground,  and  is  processed  by  the  ground  system.  The  objective  of  Telemetry  Validation 
(TV)  is  to  ensure  that  the  end  product  (the  raw,  processed,  and  displayed  telemetry) 
agrees  with  the  data  being  produced  on  the  satellite,  as  interpreted  in  accordance  with  the 
telemetry  database  provided  by  the  contractor.  The  steps  to  ensure  the  telemetry  is  valid 
are  as  follows:  (1)  Validating  the  raw  telemetry  at  the  output,  (2)  Validating  the 
processed  telemetry  products  (EU  converted  files,  etc.),  (3)  Validating  the  displayed 
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telemetry  (telemetry  screens)  is  properly  converted  by  the  ground  system  and  (4) 
Validating  that  of  red,  yellow,  and  green  limits  are  handled  correctly. 

Rehearsals:  Like  Exercises,  Rehearsals  are  primarily  used  as  training  events,  but  they 
have  a  secondary  mission  of  testing  the  ground  system  and  operational  concept  under 
realistic  loading  conditions.  Also  Rehearsals  are  often  executed  as  needed  throughout  the 
validation  process.  Because  of  this,  our  SMEs  concluded  that  they  did  not  need  to  be 
individually  called  out  in  our  process. 

Day  in  the  Eife  Tests:  Day  in  the  Eife  Test  (DITE)  is  the  only  step  in  the  validation 
process  that  was  added  based  on  SME  feedback.  The  DITE  exercises  the  system  based 
on  a  normal  day’s  activities  (not  a  Eaunch  and  Early  Orbit  (EEO)  day’s  activities.)  The 
main  goal  is  to  identify  any  deficiencies  with  the  ground  system  that  would  prevent 
normal  operations.  A  secondary  goal  is  to  examine  the  routine  operations  usability  of  the 
system  at  a  point  where  there  is  still  some  ability  to  make  modifications  if  a  more 
efficient,  or  better  process  can  be  established.  Routine  procedures  are  run,  post  pass 
processing  of  data  should  be  completed;  everything  should  work  as  expected  on-orbit  or 
changes  to  the  Concept  of  Operations  (CONOPS)  or  the  ground  system  need  to  be  made. 
The  focus  is  on  the  procedures,  CONOPS,  and  the  ground  system’s  ability  to  perform  the 
procedures  and  CONOPS.  A  DITE  is  most  successful  with  participation  from  the  satellite 
contractor,  the  ground  system  contractor  and  the  operator. 

Eaunch  Based  Compatibility  Test:  The  Eaunch  Based  Compatibility  Test  (EBCT) 
is  the  final  opportunity  for  the  ground  system  and  satellite  to  connect  prior  to  launch.  All 
of  the  objectives  accomplished  at  ECT  are  revalidated  during  the  EBCT.  In  addition  to 
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the  main  objectives  from  FCT,  LBCT  ensures  that  the  satellite  transponder  was  not 
damaged  during  the  shipment  of  the  satellite  to  the  launch  site  and  that  any  previously 
encountered  compatibility  issues  have  been  resolved.  Since  LBCT  is  the  last  chance  to 
confirm  compatibility,  the  ground  system  and  satellite  baseline  are  typically  frozen  after  a 
successful  test,  and  no  software  or  hardware  changes  are  allowed  until  after  the  satellite 
has  launched  and  is  in  a  safe  configuration.  Freezes  prevent  the  team  from  inadvertently 
changing  something  that  does  not  allow  the  ground  system  to  contact  the  satellite. 
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Appendix  C:  Survey  #1 


STEP  1:  Develop  a  Validation  Cost  Model 

For  the  scope  of  our  thesis  we  will  be  examining  the  compatibility  between  an  R&D  spacecraft  and 
its  associated  ground  system.  We  will  specifically  be  looking  at  the  process  for  validating  this 
compatibility.  Below  is  a  model  of  the  process.  With  this  survey  we  also  sent  out  this  process  with 
additional  information.  Please  use  this  model  process  for  answering  the  questions  within  our 
survey. 

Validation  Process:  For  the  Validation  process  discussed  throughout  this  survey,  the  model 
referenced,  is  the  model  shown  below. 
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Define  the  validation  process 

This  first  subsection  deals  with  the  validation  process  identified  on  page  1  of  this 
survey.  Please  review  this  process.  After  reviewing  the  process,  please  answer 
question  1-5.  If  you  answer  “NO”  to  any  question,  please  explain  in  the  comment 
section.  Please  include  any  additional  comments  you  may  have  in  this  section. 
Additional  information  about  the  process  model  is  included  in  the  PowerPoint 
presentation  sent  out  with  this  survey.  Details  include  objectives  for  each  step  in 
the  process. 

Are  these  the  right  steps  in  the  proeess? 

Comments: 

Yes  No 

Are  the  steps  in  the  right  order? 

Comments: 

Yes  No 

Is  the  process  complete? 

Comments: 

Yes  No 

If  you  answered  no  above,  what  steps  in  the  process  are  missing? 

Comments: 

Are  there  any  steps  you  don’t  feel  are  part  of  the  validation  effort? 

Comments: 

Yes  No 

Assign  appropriate  cost  to  each  activity 

This  second  subsection  deals  with  assigning  a  cost  to  each  of  the  steps  in  the 
validation  process.  Please  review  the  process  model  and  answer  questions  1-5. 
Please  incorporate  any  comments  from  above  and  include  any  comments  you  may 
have  in  the  comments  section. 
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Assign  a  cost  to  each  validation  activity. 
Please  round  to  the  nearest  $10K 

Comments: 


Operational 

Acceptance 

Testing 

Exercise  1 _ 

Data  Flow 
Testing 

Rehearsal  1 _ 

Factory  Compatibility 
Test 

Exercise  2 _ 

Week  In  The  Fife  Test 
1 

Command 

Validation 

Telemetry 

V  alidation _ 

Rehearsal  2 _ 

Week  in  the  life  test 
2 

Faunch  Based 
Compatibility 
Test 

Rehearsal  3 

Mission  Dress 
Rehears  al _ 


What  are  the  assumptions  and  limitations  used  to  assign  the  cost  for  the 
Operational  Acceptance  Testing? 

Comments: 


What  are  the  assumptions  and  limitations  used  to  assign  the  cost  for  the  Exereises? 
Comments: 


What  are  the  assumptions  and  limitations  used  to  assign  the  cost  for  the  Data  Flow 
2c.  Test? 

Comments: 


What  are  the  assumptions  and  limitations  used  to  assign  the  cost  for  the 
2d.  Rehearsals? 

Comments: 


What  are  the  assumptions  and  limitations  used  to  assign  the  cost  for  the  Factory 
2e.  Compatibility  Test 
Comments: 
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2f. 

What  are  the  assumptions  and  limitations  used  to  assign  the  cost  for  the  Week  in 
the  Life  Test? 

Comments: 

2g- 

What  are  the  assumptions  and  limitations  used  to  assign  cost  for  the  Command 
Validation? 

Comments: 

2h. 

What  are  the  assumptions  and  limitations  used  to  assign  cost  for  the  Telemetry 
Validation? 

Comments: 

2i. 

What  are  the  assumptions  and  limitations  used  to  assign  cost  for  the  Launch  Based 
Compatibility  Test? 

Comments: 

2j. 

What  are  the  assumptions  and  limitations  used  to  assign  cost  for  the  Mission  Dress 
Rehearsal? 

Comments: 

3. 

What  past  program  information  was  used  to  ascertain  this  cost? 

Comments: 

D 

What  is  the  impact  of  not  executing  each  activity? 

Comments: 

5. 

What  other  options  are  available  for  each  step? 

Comments: 
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STEP  2:  Identify  the  Risks  Associated  With  the  Compatibility  between  R&D 
Satellite  and  their  Ground  Systems,  Assign  Risks  to  Isolated  Steps  in  the 
Validation  Process  and  Define  Costs  Associated  with  Impacts  of  Risks 

The  compatibility  between  a  satellite  and  its  ground  system  is  very  important. 
Every  satellite  program  office  assesses  and  monitors  the  risks  associated  with  this 
compatibility.  Each  step  in  the  validation  process  is  performed  in  order  to  mitigate 
one  of  more  risks  associated  with  the  compatibility  between  a  satellite  and  its 
ground  system.  In  this  step,  we  would  like  to  get  your  input. 

Please  use  the  following  matrix  to  determine  the  probability  and  the  consequences 
of  the  occurrence  each  risk  identified. 


> 

!5 

(0 

£1 

O 

a 

o 

o 

£ 

0) 


Identify  the  Risks  Associated  with  the  Compatibility  between  an  R&D 
Satellite  and  its  Ground  System 

The  validation  process  is  used  to  mitigate  the  risks  associated  with  the 
compatibility  between  satellite  and  their  ground  systems.  In  this  subsection,  please 
describe  all  of  the  risks  a  typical  program  office  would  encounter  on  this  subject 


1 

Identify  the  risks  associated  with  the  compatibility  between  R&D  satellite 
and  ground  systems? 

Assign  Risks  to  Isolated  Steps  in  the  Validation  Process 

For  each  risk  identified  above,  please  answer  the  following: 

Negligible 

Minor 

Moderate 

Serious 

91-100% 

61  ■  90% 

41  ■  60% 

11-40% 

0-10% 

Critical 


Consequences/Impact 
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la. 

Assign  Risks  to  each  step  in  the  validation  process? 

For  the  first  risk  you  identified  above. 

Comments: 

lb. 

Assign  Risks  to  each  step  in  the  validation  process? 

For  the  second  risk  you  identified  above. 

Comments: 

Ic. 

Assign  Risks  to  each  step  in  the  validation  process? 

For  the  third  risk  you  identified  above. 

Comments: 

Id. 

Assign  Risks  to  each  step  in  the  validation  process? 

For  the  fourth  risk  you  identified  above. 

Comments: 

le. 

Assign  Risks  to  each  step  in  the  validation  process? 

For  the  fifth  risk  you  identified  above. 

Comments: 

If. 

Assign  Risks  to  each  step  in  the  validation  process? 

For  the  sixth  risk  you  identified  above. 

Comments: 

Ig- 

Assign  Risks  to  each  step  in  the  validation  process? 

For  the  seventh  risk  you  identified  above. 

Comments: 

Ih. 

Assign  Risks  to  each  step  in  the  validation  process? 

For  the  eighth  risk  you  identified  above. 

Comments: 

li. 

Assign  Risks  to  each  step  in  the  validation  process? 

For  the  ninth  risk  you  identified  above. 

Comments: 

Ij- 

Assign  Risks  to  each  step  in  the  validation  process? 

For  the  tenth  risk  you  identified  above. 

Comments: 

Assign  Probabilities  and  Impacts  to  Each  Risk  Identified  Above 

For  each  risk  identified  above,  please  answer  the  following: 

If  you  don’t  need  to  use  each  box,  please  leave  the  answer  blank.  If  we  did  not 
provide  enough  spaces,  please  insert  additional  rows.  If  a  risk  has  multiple  impacts. 


please  identify  each  impact  as  a  separate  line  item. 


If  none  of  the  validation  steps  are  performed,  what  is  the 

0-10% 

probability  of  the  risk  occurring? 

11-40% 

la. 

For  the  first  risk  you  identified  above. 

41-60% 

61-90% 

Comments: 

91-100% 
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If  none  of  the  validation  steps  are  performed,  what  is  the  0-10% 

probability  of  the  risk  occurring?  1 1-40% 

lb.  For  the  second  risk  you  identified  above.  41-60% 

61-90% 

Comments:  91-100% 


If  none  of  the  validation  steps  are  performed,  what  is  the  0-10% 

probability  of  the  risk  occurring?  1 1-40% 

Ic.  For  the  third  risk  you  identified  above.  41-60% 

61-90% 

Comments:  91-100% 


If  none  of  the  validation  steps  are  performed,  what  is  the  0-10% 

probability  of  the  risk  occurring?  1 1-40% 

Id.  For  the  fourth  risk  you  identified  above.  41-60% 

61-90% 

Comments:  91-100% 


If  none  of  the  validation  steps  are  performed,  what  is  the  0-10% 

probability  of  the  risk  occurring?  1 1-40% 

le.  For  the  fifth  risk  you  identified  above.  41-60% 

61-90% 

Comments:  91-100% 


If  none  of  the  validation  steps  are  performed,  what  is  the  0-10% 

probability  of  the  risk  occurring?  1 1-40% 

If.  For  the  sixth  risk  you  identified  above.  41-60% 

61-90% 

Comments:  91-100% 


If  none  of  the  validation  steps  are  performed,  what  is  the  0-10% 

probability  of  the  risk  occurring?  1 1-40% 

Ig.  For  the  seventh  risk  you  identified  above.  41-60% 

61-90% 

Comments:  91-100% 


If  none  of  the  validation  steps  are  performed,  what  is  the  0-10% 

probability  of  the  risk  occurring?  1 1-40% 

Ih.  For  the  eighth  risk  you  identified  above.  41-60% 

61-90% 

Comments:  91-100% 


If  none  of  the  validation  steps  are  performed,  what  is  the  0-10% 

probability  of  the  risk  occurring?  1 1-40% 

li.  For  the  ninth  risk  you  identified  above.  41-60% 

61-90% 

Comments:  91-100% 
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If  none  of  the  validation  steps  are  performed,  what  is  the  0-10% 

probability  of  the  risk  occurring?  1 1-40% 

li.  For  the  tenth  risk  you  identified  above.  41-60% 

61-90% 

Comments:  91-100% 


What  is  the  probability  of  the  risk  occurring  after  performing  0-10% 

the  steps  in  the  validation  process?  1 1-40% 

2a.  For  the  first  risk  you  identified  above.  41-60% 

61-90% 

Comments:  91-100% 


If  none  of  the  validation  steps  are  performed,  what  is  the  0-10% 

probability  of  the  risk  occurring?  1 1-40% 

2b.  For  the  second  risk  you  identified  above.  41-60% 

61-90% 

Comments:  91-100% 


If  none  of  the  validation  steps  are  performed,  what  is  the  0-10% 

probability  of  the  risk  occurring?  1 1-40% 

2c.  For  the  third  risk  you  identified  above.  41-60% 

61-90% 

Comments:  91-100% 


If  none  of  the  validation  steps  are  performed,  what  is  the  0-10% 

probability  of  the  risk  occurring?  1 1-40% 

2d.  For  the  fourth  risk  you  identified  above.  41-60% 

61-90% 

Comments:  91-100% 


If  none  of  the  validation  steps  are  performed,  what  is  the  0-10% 

probability  of  the  risk  occurring?  1 1-40% 

2e.  For  the  fifth  risk  you  identified  above.  41-60% 

61-90% 

Comments:  91-100% 


If  none  of  the  validation  steps  are  performed,  what  is  the  0-10% 

probability  of  the  risk  occurring?  1 1-40% 

2f.  For  the  sixth  risk  you  identified  above.  41-60% 

61-90% 

Comments:  91-100% 


If  none  of  the  validation  steps  are  performed,  what  is  the  0-10% 

probability  of  the  risk  occurring?  1 1-40% 

2g.  For  the  seventh  risk  you  identified  above.  41-60% 

61-90% 

Comments:  91-100% 
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2h. 

If  none  of  the  validation  steps  are  performed,  what  is  the 
probability  of  the  risk  occurring? 

For  the  eighth  risk  you  identified  above. 

Comments: 

0-10% 

11-40% 

41-60% 

61-90% 

91-100% 

If  none  of  the  validation  steps  are  performed,  what  is  the 

0-10% 

probability  of  the  risk  occurring? 

11-40% 

2i. 

For  the  ninth  risk  you  identified  above. 

41-60% 

61-90% 

Comments: 

91-100% 

If  none  of  the  validation  steps  are  performed,  what  is  the 

0-10% 

probability  of  the  risk  occurring? 

11-40% 

2j. 

For  the  tenth  risk  you  identified  above. 

41-60% 

61-90% 

Comments: 

91-100% 

If  the  risk  occurs,  what  are  the  impacts  associated  with  it? 
For  the  first  risk  you  identified  above. 

Comments: 

If  the  risk  occurs,  what  are  the  impacts  associated  with  it? 
For  the  second  risk  you  identified  above. 

Comments: 

If  the  risk  occurs,  what  are  the  impacts  associated  with  it? 
For  the  third  risk  you  identified  above. 

Comments: 

If  the  risk  occurs,  what  are  the  impacts  associated  with  it? 
For  the  fourth  risk  you  identified  above. 

Comments: 

If  the  risk  occurs,  what  are  the  impacts  associated  with  it? 
For  the  fifth  risk  you  identified  above. 

Comments: 

If  the  risk  occurs,  what  are  the  impacts  associated  with  it? 
For  the  sixth  risk  you  identified  above. 

Comments: 

If  the  risk  occurs,  what  are  the  impacts  associated  with  it? 
For  the  seventh  risk  you  identified  above. 

Comments: 


3a. 

3b. 

3c. 

3d. 

3e. 

3f. 

3g. 
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3h. 

If  the  risk  occurs,  what  are  the  impacts  associated  with  it? 

For  the  eighth  risk  you  identified  above. 

Comments: 

31. 

If  the  risk  occurs,  what  are  the  impacts  associated  with  it? 

For  the  ninth  risk  you  identified  above. 

Comments: 

3j. 

If  the  risk  occurs,  what  are  the  impacts  associated  with  it? 

For  the  tenth  risk  you  identified  above. 

Comments: 

4a. 

What  is  the  severity  of  each  impact? 

For  the  first  impact  you  identified  above. 

Comments 

Negligible 

Minor 

Moderate 

Serious 

Critical 

4b. 

What  is  the  severity  of  each  impact? 

For  the  second  impact  you  identified  above. 

Comments 

Negligible 

Minor 

Moderate 

Serious 

Critical 

4c. 

What  is  the  severity  of  each  impact? 

For  the  third  impact  you  identified  above. 

Comments 

Negligible 

Minor 

Moderate 

Serious 

Critical 

4d. 

What  is  the  severity  of  each  impact? 

For  the  fourth  impact  you  identified  above. 

Comments 

Negligible 

Minor 

Moderate 

Serious 

Critical 

4e. 

What  is  the  severity  of  each  impact? 

For  the  fifth  impact  you  identified  above. 

Comments 

Negligible 

Minor 

Moderate 

Serious 

Critical 

4f. 

What  is  the  severity  of  each  impact? 

For  the  sixth  impact  you  identified  above. 

Comments 

Negligible 

Minor 

Moderate 

Serious 

Critical 
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4g- 

What  is  the  severity  of  each  impact? 

For  the  seventh  impact  you  identified  above. 

Comments 

Negligible 

Minor 

Moderate 

Serious 

Critical 

4h. 

What  is  the  severity  of  each  impact? 

For  the  eighth  impact  you  identified  above. 

Comments 

Negligible 

Minor 

Moderate 

Serious 

Critical 

4i. 

What  is  the  severity  of  each  impact? 

For  the  ninth  impact  you  identified  above. 

Comments 

Negligible 

Minor 

Moderate 

Serious 

Critical 

4j. 

What  is  the  severity  of  each  impact? 

For  the  tenth  impact  you  identified  above. 

Comments 

Negligible 

Minor 

Moderate 

Serious 

Critical 

Define  costs  Associated  with  Risk  Impacts 

For  the  scope  of  this  thesis  we  are  assuming  that  each  risk  impact  has  a  cost  associated 
with  it.  Please  assign  a  cost  impact  for  each  identified  above.  If  there  are  multiple 
costs  associated  with  a  risk  impact,  please  put  the  total  down  and  explain  the  multiple 
costs  in  the  comment  section.  If  a  risk  has  multiple  impacts  please  track  each  impact 
as  a  separate  line  item. 

la. 

What  are  the  cost  associated  with  the  impacts,  if  the  risk 
occurs. 

For  the  first  risk  you  identified  above. 

Comments: 

$ 

lb. 

What  are  the  cost  associated  with  the  impacts,  if  the  risk 
occurs. 

For  the  second  risk  you  identified  above. 

Comments: 

$ 

Ic. 

What  are  the  costs  associated  with  the  impacts,  if  the  risk 
occurs. 

For  the  third  risk  you  identified  above. 

Comments: 

$ 

Id. 

What  are  the  costs  associated  with  the  impacts,  if  the  risk 
occurs. 

For  the  fourth  risk  you  identified  above. 

Comments: 

$ 
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le. 

What  are  the  costs  associated  with  the  impacts,  if  the  risk 
occurs. 

For  the  fifth  risk  you  identified  above. 

Comments: 

$ 

If. 

What  are  the  costs  associated  with  the  impacts,  if  the  risk 
occurs. 

For  the  sixth  risk  you  identified  above. 

Comments: 

$ 

Ig- 

What  are  the  costs  associated  with  the  impacts,  if  the  risk 
occurs. 

For  the  seventh  risk  you  identified  above. 

Comments: 

$ 

Ih. 

What  are  the  costs  associated  with  the  impacts,  if  the  risk 
occurs. 

For  the  eighth  risk  you  identified  above. 

Comments: 

$ 

li. 

What  are  the  costs  associated  with  the  impacts,  if  the  risk 
occurs. 

For  the  ninth  risk  you  identified  above. 

Comments: 

$ 

Ij- 

What  are  the  costs  associated  with  the  impacts,  if  the  risk 
occurs. 

For  the  tenth  risk  you  identified  above. 

Comments: 

$ 

Open  Ended  Questions 

The  following  questions  may  or  may  not  be  formally  used  in  our  research.  However, 
we  would  appreciate  additional  feedback. 

1. 

Why  is  the  compatibility  between  a  small  R&D  satellite  and  its  ground  system 
so  important? 

Comments: 

2. 

Can  we  partially  reduce  the  probability  of  a  risk  occurring  by 
performing  a  cheaper/less  extensive  variation  of  a  validation 
step?  If  so,  please  comment  on  which  step  and  how. 
Comments: 

Yes  No 

3. 

Does  a  step  in  the  validation  process  reduce  the  portability  of 
more  than  one  risk  occurring?  If  so  which  ones? 

Comments: 

Yes  No 

■ 

Are  there  any  risks  that  can  occur,  that  require  multiple 
validation  steps?  If  so  which  ones? 

Comments: 

Yes  No 
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5a. 

What  tradeoffs  are  available  in  each  of  the  risk  areas? 

For  the  first  risk  you  identified  above. 

Comments: 

5b. 

What  tradeoffs  are  available  in  each  of  the  risk  areas? 

For  the  second  risk  you  identified  above. 

Comments: 

5c. 

What  tradeoffs  are  available  in  each  of  the  risk  areas? 

For  the  third  risk  you  identified  above. 

Comments: 

5d. 

What  tradeoffs  are  available  in  each  of  the  risk  areas? 

For  the  fourth  risk  you  identified  above. 

Comments: 

5e. 

What  tradeoffs  are  available  in  each  of  the  risk  areas? 

For  the  fifth  risk  you  identified  above. 

Comments: 

5f. 

What  tradeoffs  are  available  in  each  of  the  risk  areas? 

For  the  sixth  risk  you  identified  above. 

Comments: 

5g- 

What  tradeoffs  are  available  in  each  of  the  risk  areas? 

For  the  seventh  risk  you  identified  above. 

Comments: 

5h. 

What  tradeoffs  are  available  in  each  of  the  risk  areas? 

For  the  eighth  risk  you  identified  above. 

Comments: 

5i. 

What  tradeoffs  are  available  in  each  of  the  risk  areas? 

For  the  ninth  risk  you  identified  above. 

Comments: 

5j. 

What  tradeoffs  are  available  in  each  of  the  risk  areas? 

For  the  tenth  risk  you  identified  above. 

Comments: 
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Appendix  D:  Survey  #2 


STEP  1:  Develop  a  Validation  Cost  Model 

For  the  scope  of  our  thesis  we  will  be  examining  the  compatibility  between  an  R&D  spacecraft 
and  its  associated  ground  system.  Wie  will  specifically  be  looking  at  the  process  for  validating 
this  compatibility.  Below  is  a  model  of  the  process.  With  this  survey  we  also  sent  out  this 
process  with  additional  information.  Please  use  this  model  process  for  answering  the  questions 
within  our  survey. 

Validation  Process:  For  the  Validation  process  discussed  throughout  this  survey,  the  model 
referenced,  is  the  model  shown  below. 
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Define  the  validation  process 

This  first  subsection  deals  with  the  validation  process  identified  on  page  1  of  this 
survey.  Please  review  this  process.  The  questions  that  we  asked  you  are  listed  as  Ql- 
5  and  the  responses  given  are  listed  as  Rl.x  -  R5.x.  Please  review  the  overall  teams 
assessment  and  let  us  know  whether  you  agree  or  disagree  with  their  comments.  All 
comments  were  reviewed,  as  some  were  very  similar,  we  may  have  consolidated  the 
overall  comment,  so  you  may  not  see  your  specific  comment. 


Q1  I  Are  these  the  right  steps  in  the  process' 


R1 . 1  Only  one  Week  in  the  Life  Test  is  needed 


Rehearsal  and  training  products  are  part  of  the  training  for  the 
R1 .2  operational  team  and  are  not  part  of  the  overall  ground  system 
validation 


Agree 

Disagree 


Agree 

Disagree 


I 


Additional  testing  is  required  after  each  software  drop  and  needs  to  be  Agree 


incorporated  into  the  overall  process 


Q2  Are  the  steps  in  the  right  order? 


R2. 1  Telemetry  validation  needs  to  occur  before  command  validation 


Specific  to  missions,  frequently  operational  testing  occurs  much  later 
than  you  recommend 


Command  and  telemetry  validation  should  occur  after  each  software 
drop  (as  mentioned  in  R  1.3) 


R2  4  acceptance  testing  is  not  a  separate  function,  rather  part 

of  the  entire  process. 


^  Telemetry  and  command  validation  should  be  done  in  conjunction 
with  either  data  flow  tests  and/or  with  FCT  and  or  LBCT 


R2.6  MDR  should  not  be  part  of  the  validation  effort 


„  Prefers  order  -  DT&E,  FCT,  OAT,  Ex  1,  DFT,  C&T  Validation,  Reh 
1,WITE 


Rehearsals/Exercises  have  a  tertiary  mission  of  validation,  but 
primarily  used  for  operational  training  and  thus  do  not  need  to  come 
R2.8  at  any  specific  time  in  the  process,  but  must  be  completed,  and  in  fact 
due  help  validate  the  system  (especially  if  a  simulator  is  used  for 
commanding) 


Q3  Is  the  process  complete? 


R3. 1  Add  Day  in  the  Fife  Test  (DITE) 


-  Call  Operational  Testing  System  Testing,  which  allows  you  to  leave 
AFSPC  out  of  the  testing  loop 


Q4  If  you  answered  no  above,  what  steps  in  the  process  are  missing' 


R4. 1  DT&E  is  missing 


Disagree 


Agree 

Disagree 


Agree 

Disagree 


Agree 

Disagree 


Agree 

Disagree 


Agree 

Disagree 


Agree 

Disagree 


Agree 

Disagree 


Agree 

Disagree 


Agree 

Disagree 


Agree 

Disagree 


Agree 
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Disagree 


Q5  Are  there  any  steps  you  don’t’  feel  are  part  of  the  validation  effort? 


Mission  Dress  Rehearsal  shouldn’t  be  a  part  of  the  validation  process  Agree 
because  the  ground  system  has  already  been  validated  by  this  point.  Disagree 


Only  steps  that  interface  the  ground  system  to  the  satellite 
are  required 


LBCT  should  not  be  part  of  the  validation  effort 


Rehearsals  2, 3, etc  and  exercises  2,3,  etc.  should  not  be  a 
part  of  the  validation  effort 


Launch  should  not  be  a  part  of  the  validation  effort 


Agree 

Disagree 


Agree 

Disagree 


Agree 

Disagree 


Agree 

Disagree 


Assign  appropriate  cost  to  each  activity 

This  second  subsection  deals  with  assigning  a  cost  to  each  of  the  steps  in  the 
validation  process.  Please  review  the  process  model  and  answer  questionsl-5.  Please 
review  the  costs  that  were  provided  by  the  other  subject  matter  experts  and  let  us 
know  if  you  agree  with  the  cost  or  disagree.  If  you  only  feel  comfortable 
commenting  on  costs  from  your  experience/contract,  please  indicate  that  to  us. 

The  questions  were  wrapped  together  in  this  section  to  be  easier  for  you  to  read,  and 
hopefully  eliminate  page  flipping. 


„ . What  is  the  cost  of  Operational  Acceptance  Testing? 

^  ’  What  assumptions  and  limitations  were  used  in  this  response? 


Operational  Acceptance  Testing  (OAT) 

Contract  Hours  Dollars 

Ground  System  Development  0 

Contractor  $0.00 

Operations  Contractor _ 168 _ $12,600.00 

Test  Asset _ 0 _ $0.00 

Satellite  Development  Contractor _ 0 _ $0.00 

$12,600.00 


Rl/2a. 


Assumptions  and  Limitations: 


Ground  system  is  stable,  satellite  has  good  documentation  for  ground 
system. 

The  ground  system  is  complete  to  include  MUS 

Each  test  is  only  conducted  one  time 

Based  on  operations  contractor  PE  for  STPSat-2 


Agree 

Disagree 
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Ql/2b 

What  is  the  cost  of  Exercises? 

What  assumptions  and  limitations  were  used  in  this  response? 

*If  you  complete  your  exercises  connected  to  a  simulator  that  runs  flight 
software,  they  are  used  as  a  validation  activity.  The  primary  mission  of  a 
rehearsal  is  to  train  the  MCF,  however  this  is  a  function  of  the  rehearsal. 

Rl/2b 

Exercises 

Contract 

Hours 

Dollars 

Ground  System 

Contract 

40 

$4,000.00 

Operations  Contract 

284 

$21,300.00 

Mobile  Range  Flight 

0 

$0.00 

Satellite  Developer 

0 

$0.00 

$25,300.00 

Assumptions  and  Limitations: 

•  Exercises  are  internal,  mostly  used  for  operational  training. 

•  Operations  contractor  participation  -  3  days,  8hr/day,  7  people  ~  $21K 

•  Ground  system  development  contractor  only  provide  SA  support  - 
$4K 

Agree 

Disagree 
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Ql/2c 

What  is  the  cost  of  Data  Flow  Tests? 

What  assumptions  and  limitations  were  used  in  this  response? 

Rl/2c 

Data  Flow  Testing 

Contract 

Hours 

Dollars 

Ground  System 
Contract 

702 

$70,200.00 

Operations  Contract 

252 

$18,900.00 

Mobile  Range  Flight 

0 

$0.00 

Satellite  Developer 

454 

$34,050.00 

$123,150.00 

Assumptions  and  Limitations: 

•  Ground  system/Satellite  connectivity  required,  ground  system 
and  factory  are  remote  from  one  another 

•  No  deployables  equipment  is  required  (MRF) 

•  Mobile  comm,  system  required  ($50K  per  event) 

•  Deployment  of  mobile  comm 

•  Ground  System  Communication  is  available 

•  Ground  System  Development  Contractor  Cost  covers  travel, 
setup  and  site  surveys 

•  Other  includes  SV  and  payload  TA  support 

Agree 

Disagree 

Ql/2d 

What  is  the  cost  of  Rehearsals? 

What  assumptions  and  limitations  were  used  in  this  response? 

*Ifyou  complete  your  rehearsals  connected  to  a  simulator  that  runs 
flight  software,  they  are  used  as  a  validation  activity.  The  primary 
mission  of  a  rehearsal  is  to  train  the  MCF,  however  this  is  a  function 
of  the  rehearsal. 
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Rehearsals 


Rl/2d 


Rl/2e. 


Contract 

Hours 

Dollars 

Ground  System 
Contract 

120 

$12,000.00 

Operations  Contract 

524 

$39,300.00 

Mobile  Range  Flight 

0 

$0.00 

Satellite  Developer 

1200 

$90,000.00 

$141,300.00 

Ql/2e 


Assumptions  and  Limitations: 

•  Ground  system/Satellite  connectivity  is  not  required 

•  All  critical  MCF  personnel  are  required  to  be  located  at  the 
RSC 

•  Operations  contractor  support  13  people  8  hours  a  day  for  5 
days 

•  Ground  system  development  contractor  will  provide  SA 
support  only 

•  24  hour  Ground  system  development  contractor  support  total 
~$12K 

•  Other  includes  SV  and  payload  TA  support 


Agree 

Disagree 


What  is  the  cost  of  Factory  Compatibility  Test? 

What  assumptions  and  limitations  were  used  in  this  response? 


Factory  Compatibility  Test  (FCT)  with  TSTR 
Contract  Hours  Dollars 

Ground  System  Development 


Contractor _ 

Operations  Contractor 
Test  Asset  Flight  Site  Survey 
Test  Asset  Flight  Operations 
Satellite  Development 
Contractor 


FCT  with  STGS-T 

Contract  Hours 

Ground  System  Development 

Contractor _ 296 

Operations  Contractor  412 

Test  Asset  Flight  Site  Survey 


$29,600.00 

$30,900.00 

$60,000.00 

$220,000.00 

$53,100.00 

$393,600.00 


Dollars 


$29,600.00 

$30,900.00 

$60,000.00 


no 


Test  Asset  Flight  Operations 
Satellite  Development 
Contractor 


$160,000.00 

$53,100.00 

$333,600.00 


Ql/2f 


Rl/2f. 


Assumptions  and  Limitations: 

•  Ground  System/Satellite  connectivity  required 

•  Ground  System  and  satellite  are  remote  from  one  another 

•  Tester  is  required  ($220K) 

•  Tester  Site  Survey  Required  ($60K) 

•  Mobile  comm,  required  already  in  place  from  DFT 

•  Mobile  ground  system  already  in  place,  travel  added  to  ensure 
sys  ready 

•  Other  includes  SV  and  payload  TA  support 


Agree  Disagree 


What  is  the  cost  of  WITL? 

What  assumptions  and  limitations  were  used  in  this  response? 


Week  In  The  Life  Test  (WITL) 

Contract  Hours  Dollars 

Ground  System  Development 

Contractor  40  $4,000.00 

Operations  Contractor _ 360 _ $27,000.00 

Test  Asset _ 0 _ $0.00 

Satellite  Development  Contractor  0  $0.00 

$31,000.00 


Ql/2g 


Assumptions  and  Limitations: 

•  Ground  System  Development  Contractor  SA  Support  Only 

•  Operations  contractor  internal  exercise 

•  No  one  required  to  travel 


Agree  Disagree 


What  is  the  cost  of  Command  Validation? 

What  assumptions  and  limitations  were  used  in  this  response? 


Command  Validation  (CV) 


Contract 

Hours 

Dollars 

Ground  System  Development 
Contractor 

40 

$4,000.00 

Operations  Contractor 

106 

$7,950.00 

Test  Asset 

0 

$0.00 

Satellite  Development  Contractor 

106 

$7,950.00 

ill 


$19,900.00 

Assumptions  and  Limitations: 

•  Minimal  Ground  system  development  contraetor  support 

•  Primarily  operations  contractor,  SV  Contractor  and  Government  Task 

•  Other  includes  SV  and  payload  TA  support 

Agree  Disagree 

Ql/2 

h 

What  is  the  cost  of  Telemetry  Validation? 

What  assumptions  and  limitations  were  used  in  this  response? 
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Rl/2h 

Telemetry  Validation  (TV) 

Contract 

Hours 

Dollars 

Ground  System  Development 
Contractor 

40 

$4,000.00 

Operations  Contractor 

106 

$7,950.00 

Test  Asset 

0 

$0.00 

Satellite  Development  Contractor 

106 

$7,950.00 

$19,900.00 

Assumptions  and  Limitations: 

•  Minimal  Ground  system  development  contractor  support 

•  Primarily  operations  contractor,  SV  Contractor  and  Government  Task 

•  Other  includes  SV  and  payload  TA  support 

Agree 

Disagree 

Ql/2i. 

What  is  the  cost  of  LBCT? 

What  assumptions  and  limitations  were  used  in  this  response? 

Rl/2i. 

Launch  Based  Compatibi 

ity  Test  (LBCT)  at  ARTS  Site 

Contract 

Hours 

Dollars 

Ground  System  Development 
Contractor 

280 

$28,000.00 

Operations  Contractor 

412 

$30,900.00 

Test  Asset 

0 

$0.00 

Satellite  Development  Contractor 

1000 

$75,000.00 

$133,900.00 

LBCT  with  TSTR 

Contract 

Hours 

Dollars 

Ground  System  Development 
Contractor 

296 

$29,600.00 

Operations  Contractor 

412 

$30,900.00 

Test  Asset 

$220,000.00 

Satellite  Development  Contractor 

708 

$53,100.00 

$333,600.00 

LBCT  at  Launch  Site  with  STGS-T 

Contract 

Hours 

Dollars 

Ground  System  Development 
Contractor 

280 

$28,000.00 

Operations  Contractor 

412 

$30,900.00 

Test  Asset  Operations 

0 

$160,000.00 

Test  Asset  Flight  Site  Survey 

$60,000.00 
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Satellite  Development  Contractor 


$75,000.00 

$353,900.00 


LBCT  at  Factory  with  STGS-T 


Contract 

Hours 

Dollars 

Ground  System  Development 
Contractor 

280 

$28,000.00 

Operations  Contractor 

412 

$30,900.00 

Test  Asset 

0 

$160,000.00 

Satellite  Development  Contractor 

708 

$53,100.00 

$272,000.00 

LBCT  at  Launch  Site  with  TSTR 


Contract 

Hours 

Dollars 

Ground  System  Development 
Contractor 

296 

$29,600.00 

Operations  Contractor 

412 

$30,900.00 

Test  Asset  Operations 

$220,000.00 

Test  Asset  Flight  Site  Survey 

$60,000.00 

Satellite  Development  Contractor 

1000 

$75,000.00 

$415,500.00 

Assumptions  and  Limitations: 

•  If  launched  somewhere  w/o  AFSCN,  LBCT  will  be  more 

•  If  launched  from  either  Cape  Canaveral  or  Vandenberg,  cost  is  minimal 

•  Ground  system  development  contractor  assumed  no  travel  required  and 
launch  is  at  Cape  or  Vandenberg 

•  Other  includes  SV  and  payload  TA  support 


Agree 

Disagree 


01/2’  MDR? 

^  ^  What  assumptions  and  limitations  were  used  in  this  response? 


Assumptions  and  Limitations: 

•  Ground  System/Satellite  connectivity  not  required 
Rl/2j  •  Full  LEO  capability 

•  3  days,  24  hr/day 

•  Ground  system  development  contractor  will  provide  SA  support  only 


Agree 

Disagree 


Q3.  What  past  program  information  was  used  to  ascertain  this  cost? 
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*we  do  not  feel  this  question  required  response  on  subsequent  surveys  and  only  found  it 
helpful  for  our  knowledge. 


Q4. 

What  is  the  impact  of  not  executing  each  activity? 

R4a 

OAT-  Ground  system  would  not  be  validated  and  deemed 
acceptable  to  conduct  operations  meeting  all  requirements. 

Agree 

Disagree 

R4b 

Exercises  -  Exercises  provide  a  means  to  indentify  project 
deficiencies  related  to  the  ground  system  or  mission  planning 
processes;  without  these  events  potential  impacts  to  the  project 
schedule  and  cost  exist  due  to  the  discovery  of  the  deficiencies 
later  in  the  project  schedule. 

Agree 

Disagree 

R4c 

Data  Elow  Testing  -  Issues  with  ground  system  to  satellite 
compatibility  would  not  be  identified  at  earliest  opportunity. 

Agree 

Disagree 

R4d 

Rehearsals  -  Inadequate  preparedness  of  operations  support 
staff  to  perform  mission  operations;  unfamiliarity  with  the 
ground  system  being  used  to  conduct  operations.  Operational 
impacts  to  functionality  of  ground  system  would  not  be 
assessed. 

Agree 

Disagree 

R4e 

ECT  -  Inability  to  verify  correct  ARTS  IRON  database 
configuration;  could  potentially  result  in  loss  of  mission. 

Agree 

Disagree 

R4f 

WITE  testing  -  Conducted  to  identify  any  shortcomings  with 
data  processing  over  an  extended  period  of  time  and  to  assess 
the  ground  system  stability  over  an  extended  period  of  time. 

Eor  some  missions  this  event  has  not  been  conducted  without 
impact  to  the  project. 

Agree 

Disagree 

R4h 

Command  validation  -  Significant  risk  of  inability  to  properly 
command  the  satellite;  could  result  in  loss  of  mission  or  data. 

Agree 

Disagree 

R4g 

Telemetry  validation  -  Inability  to  adequately  assess  the  health 
and  safety  of  the  satellite;  could  result  in  degraded  performance 
or  loss  of  mission. 

Agree 

Disagree 

R4i 

EBCT  -  satellite  could  have  been  damaged  during  transport  to 
the  launch  facility;  could  result  in  loss  of  mission. 

Agree 

Disagree 

R4j 

MDR  -  Validation  that  mission  operations  team  is  prepared  to 
support  the  satellite  once  on-orbit;  failure  to  conduct  this  event 
could  result  in  launching  with  a  support  staff  that  is  unprepared 
for  launch. 

Agree 

Disagree 

Q5 

What  other  options  are  available  for  each  step? 

R5 

May  be  feasible  to  add  more  steps 

Agree 

Disagree 

STEP  2: 

Identify  the  Risks  Associated  With  the  Compatibility  between  R&D  Satellite  and  their 
Ground  Systems,  Assign  Risks  to  Isolated  Steps  in  the  Validation  Process  and  Define 
Costs  Associated  with  Impacts  of  Risks 

The  compatibility  between  a  satellite  and  its  ground  system  is  very  important.  Every 
satellite  program  office  assesses  and  monitors  the  risks  associated  with  this 
compatibility.  Each  step  in  the  validation  process  is  performed  in  order  to  mitigate  one 
of  more  risks  associated  with  the  compatibility  between  a  satellite  and  its  ground 
system.  In  this  step,  we  would  like  to  get  your  input. 

Please  use  the  following  matrix  to  determine  the  probability  and  the  consequences  of  the 
occurrence  each  risk  identified. 


n 

a 

n 

o 


T3 

o 

o 

£ 

0) 


Consequences/Impact 


Identify  the  Risks  Associated  with  the  Compatibility  between  an  R&D  Satellite  and  its 
Ground  System 

The  validation  process  is  used  to  mitigate  the  risks  associated  with  the  compatibility 
between  satellite  and  their  ground  systems.  In  this  subsection,  please  describe  all  of  the 
risks  a  typical  program  office  would  encounter  on  this  subject 

All  questions  in  this  section  were  wrapped  into  one.  Therefore  the  new  question  will  be 
spelt  out  at  the  beginning  and  is  the  same  question  for  each  risk. 
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Q 

•  Identify  the  risks  associated  with  the  compatibility  between  R&D 
satellite  and  ground  systems? 

•  Assign  Risks  to  each  Step  in  the  Validation  Process 

•  If  none  of  the  validation  steps  are  performed,  what  is  the  probability  of 
the  risk  occurring? 

•  What  is  the  probability  of  the  risk  occurring  after  performing  the  steps  in 
the  validation  process? 

•  If  the  risk  occurs,  what  are  the  impacts  associated  with  it? 

•  What  is  the  severity  of  each  impact? 

•  What  are  the  cost  associated  with  the  impacts,  if  the  risk  occurs. 

Each  Risk  below  addresses  the  question  above,  the  main  idea  is  in  bold 

Risk  1 

Risk:  RF  Compatibility  between  SC  and  GS 

Steps:  FCT  and  LBCT 

Risk  before  validation  step:  41-60% 

Risk  after  validation  step:  0-10% 

Severity:  Critical  -  unable  to  cmd,  possible  loss  of  range, 
range  rate  or  telem  data 

Cost  associated  with  risk:  Possible  loss  of  msn  -  $100M 

Agree  Disagree 
Agree  Disagree 
Agree  Disagree 
Agree  Disagree 
Agree  Disagree 

Agree  Disagree 

Risk  2 

Risk:  Configuration  incompatibility  between  RTS  &  SC 
(i.e.,  ARTS  configuration,  IRON  Database) 

Steps:  FCT  and  LBCT 

Risk  before  validation  step:  41-60% 

Risk  after  validation  step:  0-10% 

Severity:  Critical  -  unable  to  cmd,  possible  loss  of  range, 
range  rate  or  tlm  data 

Cost  associated  with  risk:  Possible  loss  of  msn  -  $100M 

Agree  Disagree 

Agree  Disagree 
Agree  Disagree 
Agree  Disagree 
Agree  Disagree 

Agree  Disagree 

Risk  3 

Risk:  Cmd  incompabilities  and  errors  between  SC  &GS 
(i.e.,  GS  cmd  database  problems) 

Steps:  Command  validation 

Risk  before  validation  step:  41-60% 

Risk  after  validation  step:  0  -10% 

Severity:  Serious  -  some  cmds  may  not  work  properly  or 
at  all 

Cost  associated  with  risk:  $1M 

Agree  Disagree 

Agree  Disagree 
Agree  Disagree 
Agree  Disagree 
Agree  Disagree 

Agree  Disagree 

Risk  4 

Risk:  Telemetry  incompatibilities  and  errors  between  SC 
and  GS  (i.e.,  GS  telemetry  database  problems) 

Steps:  Telemetry  Validation 

Risk  before  validation  step:  41-60% 

Agree  Disagree 

Agree  Disagree 
Agree  Disagree 
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Risk  6 


Risk  8 


Risk  after  validation  step:  0-10% 

Severity:  Serious  -  some  telemetry  may  be  reported 
incorrectly,  limits  may  be  set  incorrectly 

Cost  associated  with  risk:  $1M 

Agree  Disagree 
Agree  Disagree 

Agree  Disagree 

Risk:  Ground  system  software  does  not  process  and 
display  satellite  telemetry  correctly 

Steps:  Telemetry  Validation,  FCT,  DFT 

Risk  before  validation  step:  41-60% 

Risk  after  validation  step:  0-10% 

Severity:  Serious 

Cost  associated  with  risk: 

Agree  Disagree 

Agree  Disagree 
Agree  Disagree 
Agree  Disagree 
Agree  Disagree 
Agree  Disagree 

Risk:  Ground  system  software  does  not  construct  and 
release  satellite  command  correctly 

Steps:  Command  Validation,  FCT,  DFT 

Risk  before  validation  step:  11-20% 

Risk  after  validation  step:  0-10% 

Severity: 

Cost  associated  with  risk: 

Agree  Disagree 
Agree  Disagree 
Agree  Disagree 
Agree  Disagree 
Agree  Disagree 
Agree  Disagree 

Risk:  Ground  system  is  unable  correctly  post-pass  process 
payload/mission  data  correctly 

Steps:  WITL,  FCT 

Risk  before  validation  step:  41-60% 

Risk  after  validation  step:  0-10% 

Severity:  Moderate 

Cost  associated  with  risk: 

Agree  Disagree 
Agree  Disagree 
Agree  Disagree 
Agree  Disagree 
Agree  Disagree 
Agree  Disagree 

Risk:  Operational  or  data  latency  impacts  based  on 
relationship  between  ground  system  and  satellite  flight 
software  (may  add  more  complexity  requiring  more  time 
or  more  resources  based  on  flight  software  handling  of 
data) 

Steps:  Exercises  and  Rehearsals 

Risk  before  validation  step:  41-60% 

Risk  after  validation  step:  1 1-20% 

Severity:  Serious 

Cost  associated  with  risk: 

Agree  Disagree 

Agree  Disagree 
Agree  Disagree 
Agree  Disagree 
Agree  Disagree 
Agree  Disagree 

Risk:  A  satellite  manufacture  trying  something  new  with 
command,  format  which  causes  compatibility  problems 
between  ground  system  and  satellite. 

Steps:  DFT,  FCT,  LBCT,  Command  Validation, 

Telemetry  Validation 

Risk  before  validation  step:  61-80% 

Risk  after  validation  step:  1 1-20% 

Agree  Disagree 

Agree  Disagree 

Agree  Disagree 
Agree  Disagree 
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Risk  1 1 


Risk  13 


Risk  14 


Severity:  Serious  -  satellite  will  make  numberous 
changes,  adding  cost  and  schedule  (MUS  development) 
Cost  associated  with  risk:  $60-$200K  (MUS  dev  &  test 
-  Dev  $50-$150K,  Test  $10-$50K) 

Agree  Disagree 

Agree  Disagree 

Risk:  Documentation  maturity  on  satellite  -  could  have  a 
great  satellite,  but  documentation  could  be  lacking 

Steps:  DFT 

Risk  before  validation  step:  61-80% 

Risk  after  validation  step:  1 1-20% 

Severity:  Minor  -  GS  will  be  built  poorly,  and  then  will 
require  cmd  processing  and  rework,  adding  cost  and 
schedule 

Cost  associated  with  risk:  $24K  -  dev  $20K  ,  test  $4K 

Agree  Disagree 

Agree  Disagree 
Agree  Disagree 
Agree  Disagree 
Agree  Disagree 

Agree  Disagree 

Risk:  Documentation  maturity  on  GS 

Steps:  DFT 

Risk  before  validation  step:  11-20% 

Risk  after  validation  step:  0-10% 

Severity:  Minor  -  satellite  manufacture  will  build 
capability  that  ground  system  can’t  handle,  ground  system 
will  need  to  be  fixed.  Adds  cost  and  schedule 

Cost  associated  with  risk:  $24K  -  dev  $20K  ,  test  $4K 

Agree  Disagree 
Agree  Disagree 
Agree  Disagree 
Agree  Disagree 
Agree  Disagree 

Agree  Disagree 

Risk:  Maturity  of  Satellite  Development 

Steps:  Command  Validation  and  Telemetry  Validation 

Risk  before  validation  step:  11-20% 

Risk  after  validation  step:  0-10% 

Severity:  Serious  -  less  mature  satellite  is  more  likely  to 
have  changes  resulting  in  changes  to  the  ground  system 
Cost  associated  with  risk:  $60-$200K  (MUS  dev  &  test  - 
Dev  $50-$150K,  Test  $10-$50K) 

Agree  Disagree 
Agree  Disagree 
Agree  Disagree 
Agree  Disagree 
Agree  Disagree 

Agree  Disagree 

Risk:  Maturity  of  Ground  System 

Steps:  FCT,  LBCT,  DFT 

Risk  before  validation  step:  11-20% 

Risk  after  validation  step:  0-10% 

Severity:  Moderate  -  Ground  System  may  not  meet 
Satellite  schedule 

Cost  associated  with  risk:  $25K-$100K  (dev  -  $20K- 
$80K,  test  $5-$20K 

Agree  Disagree 
Agree  Disagree 
Agree  Disagree 
Agree  Disagree 
Agree  Disagree 

Agree  Disagree 

Risk:  Lack  of  satellite  with  telemetry  truth  data 

Steps:  DFT 

Risk  before  validation  step:  41-60% 

Risk  after  validation  step:  1 1-20% 

Severity:  Minor  -  telemetry  not  processed  correctly  - 
rework  required  after  FCT  adding  schedule  and  cost 

Agree  Disagree 
Agree  Disagree 
Agree  Disagree 
Agree  Disagree 
Agree  Disagree 
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Cost  associated  with  risk:  $6K  (fix  database  -  $5K  + 

$1K  test) 

Agree  Disagree 

Risk  15 

Risk:  Lack  of  satellite  command  samples 

Steps:  DFT 

Risk  before  validation  step:  41-60% 

Risk  after  validation  step:  1 1-20% 

Severity:  Minor  -  commands  not  processed  correctly  - 
rework  required  after  FCT  adding  schedule  and  cost 

Cost  associated  with  risk:  $6K  (fix  -  $5K  -i-  $1K  test) 

Agree  Disagree 
Agree  Disagree 
Agree  Disagree 
Agree  Disagree 
Agree  Disagree 
Agree  Disagree 

Risk  16 

Risk:  System  fails  to  support  all  operational  requirements 
of  the  satellite 

Steps:  Exercises,  Rehearsals,  WIFE 

Risk  before  validation  step:  11-20% 

Risk  after  validation  step:  0-10% 

Severity:  Moderate  -late  fix  -  schedule  slip 

Cost  associated  with  risk: 

Agree  Disagree 
Agree  Disagree 
Agree  Disagree 
Agree  Disagree 
Agree  Disagree 
Agree  Disagree 

Risk  17 

Risk:  System  will  lose/corrupt  data 

Steps:  Command  &  Telemetry  Validation 

Risk  before  validation  step:  41-60% 

Risk  after  validation  step:  0-10% 

Severity:  Minor  -  satellite  commanding  and  telemetry  will 
be  erratic 

Cost  associated  with  risk: 

Agree  Disagree 
Agree  Disagree 
Agree  Disagree 
Agree  Disagree 
Agree  Disagree 

Agree  Disagree 

Risk  18 

Risk:  Delivered  products  will  be  improperly  formatted 
(i.e.  tasking  files) 

Steps:  Telemetry  Validation 

Risk  before  validation  step:  0-10% 

Risk  after  validation  step:  0-10% 

Severity:  Minor  -  results  in  increased  ops  costs,  replan 
contacts,  retransmit  commands,  increased  maintenance 
costs,  etc. 

Cost  associated  with  risk: 

Agree  Disagree 
Agree  Disagree 
Agree  Disagree 
Agree  Disagree 
Agree  Disagree 

Agree  Disagree 
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Appendix  E:  Survey  #3 


STEP  1:  Develop  a  Validation  Cost  Model 

For  the  scope  of  our  thesis  we  will  be  examining  the  compatibility  between  an  R&D 
satellite  and  its  associated  ground  system.  We  will  specifically  be  looking  at  the 
process  for  validating  this  compatibility.  Below  is  a  model  of  the  process.  With  this 
survey  we  also  sent  out  this  process  with  additional  information.  Please  use  this 
model  process  for  answering  the  questions  within  our  survey. 

Validation  Process:  THIS  IS  THE  ORIGINAL  VALIDATION  PROCESS.  THE 
EINAL  PROCESS  WILL  BE  CHANGED  BASED  ON  ALL  THE  INPUTS. 

Eor  the  Validation  process  discussed  throughout  this  survey,  the  model  referenced,  is 
the  model  shown  below. 
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Define  the  validation  process 

At  this  point  in  the  survey  process,  we  eliminated  all  questions/responses  that  reached 
a  consensus.  Therefore  only  responses  that  we  are  looking  for  more  insight  have 
been  included.  The  overall  comments  for/against  have  been  included.  Please  respond 
to  the  comments  in  the  right  column. 


Are  the  steps  in  the  right  order? 


Operational  acceptance  testing  is  not  a  separate  function,  rather  part  of 
the  entire  process. 


Response: 


Comment:  If  you  wait  for  rehearsals  you  may  be  too  late  and  this  is  why 
you  need  upfront  testing. 

Do  you  agree  with  this?  If  you  disagree,  please  note  why. 


Prefers  order  -  DT&E,  FCT,  OAT,  Ex  1,  DET,  C&T  Validation,  Reh  1, 
WIPE 

Comment:  Though  this  order  is  preferred  by  some,  the  actual  order  of 
validation  testing  will  vary  by  mission  and  the  availability  of  personnel, 
assets,  etc. 

Do  you  agree  with  this  belief? 


Response: 


Is  the  process  complete? 


Add  Day  in  the  Eife  Test  (DITE) 

Why  do  you  feel  a  DITE  would  be  helpful? 


If  you  answered  no  above,  what  steps  in  the  process  are  missing' 


DT&E  is  missing 

Comment:  DT&E  is  part  of  verification  and  not  validation. 


Response: 


Response: 


Do  you  agree  with  this  comment: 


Assign  appropriate  cost  to  each  activity 

This  second  subsection  deals  with  assigning  a  cost  to  each  of  the  steps  in  the 
validation  process.  Please  review  the  process  model  and  answer  questions  1-5.  Please 
review  the  costs  that  were  provided  by  the  other  subject  matter  experts  and  let  us 
know  if  you  agree  with  the  cost  or  disagree.  If  you  only  feel  comfortable 
commenting  on  costs  from  your  experience/contract,  please  indicate  that  to  us. 

The  questions  were  wrapped  together  in  this  section  to  be  easier  for  you  to  read,  and 
hopefully  eliminate  page  flipping. 

Do  you  agree  with  the  statements  the  other  experts  made  throughout  this  section  and 
why? 

01/2f  WITL? 

^  What  assumptions  and  limitations  were  used  in  this  response? _ 


Week  In  The  Life  Test  (WITL) 


Contract 

Hours 

Dollars 

Ground  System  Development  Contractor 

40 

$4,000.00 

Operations  Contractor 

360 

$27,000.00 

Rl/2f. 

Test  Asset 

0 

$0.00 

Satellite  Development  Contractor 

0 

$0.00 

$31,000.00 

Assumptions  and  Limitations: 

•  Ground  system  development  contractor  SA  Support  Only 

•  Operations  contractor  internal  exercise 


•  No  one  required  to  travel 

Comment:  Suspect  this  is  on  the  low  side.  Although  systems 

and  payload  specialists  may  not  be  required  to  travel  to  the  RSC,  Response: 

they  may  still  be  required  to  support  from  their  home  locations. 

If  you  believe  this  is  low,  please  state  a  better  estimate.  If  you 

agree  with  this  statement  but  don’t  have  an  estimate,  please  state 

as  such.  If  you  disagree,  please  tell  us  why. _ 


„ . What  is  the  cost  of  Command  Validation? 

^  ^  What  assumptions  and  limitations  were  used  in  this  response? 


123 


Rl/2g 

Command  Validation  (CV)  | 

Contract 

Hours 

Dollars 

Ground  System  Development  Contraetor 

40 

Operations  Contraetor 

106 

s 

Test  Asset 

0 

Satellite  Development  Contractor 

106 

s 

SI 

•  Minimal  Ground  System  Development  Contraetor  Support 

•  Primarily  operations  eontraetor,  SV  Contraetor  and  Government  Task 

•  Other  ineludes  SV  and  payload  TA  support 

Comment:  Insuffieient  data  to  eompute,  and 
wouldn’t  this  be  accomplished  in  conjunction  with 
some  other  event  that  eonneets  the  GS  &  SV,  or  is  a 
simulator  being  used,  and  do  you  trust  simulators? 
What  is  the  fidelity  of  the  simulator? 

Response: 

Q4. 

What  is  the  impact  of  not  executing  each  activity? 

R4e 

FCT  -  Inability  to  verify  eorreet  ARTS  IRON  database 
eonfiguration;  eould  potentially  result  in  loss  of  mission. 

There  was  some  disagreement  on  this.  Is  there  another 
impaet  that  we  are  missing?  Is  this  not  an  impaet? 

Response: 

LBCT  -  satellite  eould  have  been  damaged  during 
transport  to  the  launeh  faeility;  eould  result  in  loss  of 
mission. 

Response: 

1 

R4i 

Comment:  Although  an  LBCT  might  not  be  performed 
with  AFSCN  RTS  resources,  the  SV  manufaeturer  will 
verify  the  health  and  status  of  the  SV  at  the  launch  site 
using  the  Faetory  Ground  Support  Equipment. 

Do  you  agree  with  the  comment  above?  Are  there  other 
impaets? 

1 
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STEP  2:  Identify  the  Risks  Associated  With  the  Compatihility  between  R&D 
Satellite  and  their  Ground  Systems,  Assign  Risks  to  Isolated  Steps  in  the 
Validation  Process  and  Define  Costs  Associated  with  Impacts  of  Risks 

The  compatibility  between  a  satellite  and  its  ground  system  is  very  important.  Every 
satellite  program  office  assesses  and  monitors  the  risks  associated  with  this 
compatibility.  Each  step  in  the  validation  process  is  performed  in  order  to  mitigate  one 
of  more  risks  associated  with  the  compatibility  between  a  satellite  and  its  ground 
system.  In  this  step,  we  would  like  to  get  your  input.  . 

Please  use  the  following  matrix  to  determine  the  probability  and  the  consequences  of 
the  occurrence  each  risk  identified. 


n 

a 

n 

o 


T3 

o 

o 

£ 

0) 


Consequences/Impact 


Identify  the  Risks  Associated  with  the  Compatibility  between  an  R&D  Spacecraft  and 
its  Ground  System 

The  validation  process  is  used  to  mitigate  the  risks  associated  with  the  compatibility 
between  spacecraft  and  their  ground  systems.  In  this  subsection,  please  describe  all  of 
the  risks  a  typical  program  office  would  encounter  on  this  subject 

All  questions  in  this  section  were  wrapped  into  one.  Therefore  the  new  question  will  be 
spelt  out  at  the  beginning  and  is  the  same  question  for  each  risk. 
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•  Identify  the  risks  associated  with  the  compatibility  between  R&D 
satellite  and  ground  systems? 

•  Assign  Risks  to  each  Step  in  the  Validation  Process 

•  If  none  of  the  validation  steps  are  performed,  what  is  the  probability  of 
the  risk  occurring? 

•  What  is  the  probability  of  the  risk  occurring  after  performing  the  steps 
in  the  validation  process? 

•  If  the  risk  occurs,  what  are  the  impacts  associated  with  it? 

•  What  is  the  severity  of  each  impact? 

•  What  are  the  cost  associated  with  the  impacts,  if  the  risk  occurs. 


Risk  1:  RF  Compatibility  between  SC  and  GS 

Steps:  FCT  and  LBCT  -  Are  any  missing?  Should  some  be  added? 

Risk  before  validation  step:  41-60%  Is  this  too  high/low?  Why? _ 

Risk  2:  Configuration  incompatibility  between  RTS  &  SC  (i.e.,  ARTS  configuration, 
IRON  Database) 

Steps:  FCT  and  LBCT  -  Are  any  missing?  Should  some  be  deleted? 

Risk  before  validation  step:  41-60%  Is  this  too  high/low?  Why? 

Severity:  Critical  -  unable  to  cmd,  possible  loss  of  range,  range  rate  or  tlm  data.  If 
you  disagree  why? 

Cost  associated  with  risk:  Possible  loss  of  msn  -  $100M  -  If  you  don’t  agree  with  this 
number,  what  do  you  believe  the  risk  is? 

Risk  3:  Cmd  incompabilities  and  errors  between  SC  &GS  (i.e.,  GS  cmd  database 
problems) 

Cost  associated  with  risk:  $1M  -  If  you  disagreed,  why? _ 

Risk  4:  Telemetry  incompatibilities  and  errors  between  SC  and  GS  (i.e.,  GS  telemetry 
database  problems) 

Cost  associated  with  risk:  $1M  -  If  you  disagreed,  why? _ 

Risk  5:  Ground  system  software  does  not  process  and  display  satellite  telemetry 
correctly 

Risk  before  validation  step:  41-60%  -  Is  this  too  high/low?  Why? 

Severity:  Serious  -  If  you  disagreed,  why? _ 

Risk  6:  und  system  software  does  not  construct  and  release  satellite  command 
correctly 

Risk  before  validation  step:  1 1-20%  -  Is  this  too  high/low?  Why? _ 

Risk  7:  Ground  system  is  unable  correctly  post-pass  process  payload/mission  data 
correctly 

Risk  before  validation  step:  41-60%  -  Is  this  too  high/low?  Why? _ 

Risk  8:  Operational  or  data  latency  impacts  based  on  relationship  between  ground 
system  and  satellite  flight  software  (may  add  more  complexity  requiring  more  time  or 
more  resources  based  on  flight  software  handling  of  data) 

Steps:  Exercises  and  Rehearsals  -  Are  any  missing?  Should  some  be  deleted? _ 
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Risk  before  validation  step:  41-60%  -  Is  this  too  high/low?  Why? 
Risk  after  validation  step:  1 1-20%  -  Is  this  too  high/low?  Why? 
Severity:  Serious  -  Is  this  too  high/low?  Why? 


Risk  9:  A  satellite  manufacture  trying  something  new  with  command,  format  which 
causes  compatibility  problems  between  ground  system  and  satellite. 

Risk  after  validation  step:  1 1-20%  Should  validation  steps  reduce  this  further? 
thisfurther?WhyAVhy  not. 


Risk  10:  Documentation  maturity  on  vehicle  -  could  have  a  great  vehicle,  but  docume: 
lacking 

Steps:  DFT  -  Should  -r  FCT  -i-  LBCT  be  added? 

Risk  after  validation  step:  1 1-20%  -  Should  this  risk  be  further  reduced? 

WhyAVhy  not. 

Severity:  Minor  -  GS  will  be  built  poorly,  and  then  will  require  cmd  processing  and 
ework,  adding  cost  and  schedule  -  does  this  seem  too  low  of  a  risk  for  the  impact? 


Risk  11:  Documentation  maturity  on  GS 

Steps:  DFT  -  Are  any  missing?  Should  some  be  deleted? 

Risk  before  validation  step:  1 1-20%  -  Is  this  too  high/low?  Why? 

Risk  after  validation  step:  0-10%  -  Is  this  too  low?  Why? 

Cost  associated  with  risk:  $24K  -  dev  $20K,  test  $4K  -  Is  this  too  high/low?  Why? 


Risk  12:  Maturity  of  Vehicle  Development 

Steps:  Command  Validation  and  Telemetry  Validation  -  Are  any  missing? 
Should  some  be  deleted? 

Risk  before  validation  step:  1 1-20%  -  Is  this  too  high/low?  Why? 

Risk  after  validation  step:  0-10%  -  Is  this  too  low?  Why? 


Risk  13:  Maturity  of  Ground  System 

Risk  before  validation  step:  1 1-20%  -  is  this  too  high/low?  Why? 
Risk  after  validation  step:  0-10%  -  is  this  too  low? 


Risk  14:  Lack  of  vehicle  with  telemetry  truth  data 
Steps:  DFT  -  should  Tim  Val,  FCT,  LBCT  be  added? 

Risk  after  validation  step:  1 1-20%  -  is  this  too  high/low?  Why? 

Severity:  Minor  -  telemetry  not  processed  correctly  -  rework  required  after 
FCT  adding  schedule  and  cost  -  Is  this  the  incorrect  impact  or  severity  or  both? 


Risk  15:  Lack  of  vehicle  command  samples 

Steps:  DFT  -  Are  any  missing?  Should  some  be  deleted? 

Risk  before  validation  step:  41-60%  -  is  this  too  high/low?  Why? 

Risk  after  validation  step:  1 1-20%  -  is  this  too  high/low?  Why? 

Severity:  Minor  -  commands  not  processed  correctly  -  rework  required  after 
FCT  adding  schedule  and  cost  -  Is  this  the  incorrect  impact  or  severity  or  both? 


Risk  16:  System  fails  to  support  all  operational  requirements  of  the  satellite 
Risk  before  validation  step:  1 1-20%  -  is  this  too  high/low?  Why? 

Risk  after  validation  step:  0-10%  -  is  this  too  low?  Why? 
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Risk  17:  System  will  lose/corrupt  data 

Steps:  Command  &  Telemetry  Validation  -  are  others  missing?  Which  ones? 

Risk  after  validation  step:  0-10%  -  is  this  too  low?  Why? 

Severity:  Minor  -  satellite  commanding  and  telemetry  will  be  erratic  -  Is  this  too  low? 

V 

Risk  18:  Delivered  products  will  be  improperly  formatted  (i.e.  tasking  files) 

Steps:  Telemetry  Validation  -  should  we  add:  WITL,  ex,  reh,  and  tasking  files  for 
commanding 

Risk  before  validation  step:  0-10%  -  is  this  too  low?  Why? 
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Appendix  F:  Results  of  Survey  #1 


We  received  results  for  Survey  #1  from  the  eight  people:  three  military  members,  one 
government  civilian,  one  operations  contractor,  two  ground  system  contractors,  and  one 
independent  technical  advisor. 

Step  la:  The  Validation  Process: 

This  first  subsection  of  our  survey  dealt  with  the  validation  process  identified  on  page  in 
chapter  three.  This  section  consisted  of  five  questions.  Below  are  these  questions  and  the 
responses. 

Question  1:  Are  these  the  right  steps  in  the  process? 

Responses:  Most  of  the  SMEs  agreed  that  the  steps  in  the  process  were  correct.  Three  of 
the  SMEs  answered  no  and  provided  the  following  responses: 

1 .  Only  one  Week  in  the  Eife  Test  is  needed 

2.  Rehearsal  and  training  products  are  part  of  the  training  for  the  operational  team  and 
are  not  part  of  the  overall  ground  system  validation 

3.  Additional  testing  is  required  after  each  software  drop  and  needs  to  be  incorporated 
into  the  overall  process. 

Question  2:  Are  the  steps  in  the  right  order? 

Responses:  All  of  the  SMEs  had  comments  on  the  order  of  the  process,  many  of  these 
comments  contradicted  each  other.  We  are  hoping  to  get  greater  concurrence  on  Survey 
#2.  The  responses  were  as  follows: 

1 .  Telemetry  validation  needs  to  occur  before  command  validation 

2.  Specific  to  missions,  frequently  operational  testing  occurs  much  later  than 
recommended 

3.  Command  and  telemetry  validation  should  occur  after  each  software  drop 

4.  Operational  acceptance  testing  is  not  a  separate  function,  rather  part  of  the  entire 
process 

5.  Telemetry  and  command  validation  should  be  done  in  conjunction  with  either  data 
flow  tests  and/or  with  ECT  and  or  EBCT 

6.  MDR  should  not  be  part  of  the  validation  effort 

7.  Prefers  order  -  DT&E,  ECT,  OAT,  Ex  1,  DET,  C&T  Validation,  Reh  1,  WITE 

8.  Rehearsals/Exercises  have  a  tertiary  mission  of  validation,  but  primarily  used  for 
operational  training  and  thus  do  not  need  to  come  at  any  specific  time  in  the  process, 
but  must  be  completed,  and  in  fact  due  help  validate  the  system  (especially  if  a 
simulator  is  used  for  commanding) 

Question  3  and  4:  Is  the  process  complete?  If  you  answered  no  above,  what  steps  in  the 
process  are  missing? 
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Responses:  Most  of  the  SMEs  agreed  that  the  process  was  complete.  However,  two  of  the 
SMEs  answered  no  and  provided  the  following  responses: 

1 .  Add  Day  in  the  Eife  Test  (DITE) 

2.  Call  Operational  Testing:  System  Testing,  which  allows  you  to  leave  AESPC  out  of 
the  testing  loop 

3.  DT&E  is  missing 

Question  5:  Are  there  any  steps  you  don’t  feel  are  part  of  the  validation  effort? 

Responses:  About  half  of  the  SMEs  felt  there  were  no  steps  that  were  not  part  of  the 
validation  process.  However  the  other  half  felt  there  were  steps  that  were  unnecessary, 
they  provided  the  following  responses: 

1 .  Mission  Dress  Rehearsal  shouldn’t  be  a  part  of  the  validation  process  because  the 
ground  system  has  already  been  validated  by  this  point 

2.  Only  steps  that  interface  the  ground  system  to  the  satellite  are  required 

3.  EBCT  should  not  be  part  of  the  validation  effort 

4.  Rehearsals  2,  3,  etc  and  exercises  1,  2,  etc.  should  not  be  a  part  of  the  validation  effort 

5.  Eaunch  should  not  be  a  part  of  the  validation  effort. 

Overall  the  SMEs  agreed  that  the  validation  process  was  basically  complete.  We  did 
receive  some  comments  that  may  drive  changes  in  the  validation  process,  but  we  won’t 
make  any  of  these  changes  until  after  the  responses  are  confirmed  by  a  majority  of  the 
SMEs. 

Step  lb:  Assign  appropriate  cost  to  each  activity:  This  second  subsection  deals  with 
assigning  a  cost  to  each  of  the  steps  in  the  validation  process.  Eor  this  section,  we  did  not 
get  very  many  responses.  Because  of  this  we  used  data  from  past  programs  to  ascertain 
costs  estimate  for  each  of  the  steps  in  the  validation  process.  We  included  these  cost 
estimates  in  the  second  survey  and  are  hoping  to  get  concurrence,  or  identification  of 
problems  with  the  costs  estimates.  Below  are  the  cost  estimates  we  derived: 


Operational  Acce] 

ptance  Testing  (OAT) 

Contract 

Hours 

Dollars 

Ground  System  Development 
Contractor 

0 

$0.00 

Operations  Contractor 

168 

$12,600.00 

Test  Asset 

0 

$0.00 

Satellite  Development  Contractor 

0 

$0.00 

$12,600.00 

130 


Exercises 

Contract 

Hours 

Dollars 

Ground  System  Development 
Contractor 

40 

$4,000.00 

Operations  Contractor 

284 

$21,300.00 

Test  Asset 

0 

$0.00 

Satellite  Development  Contractor 

0 

$0.00 

$25,300.00 

Data  Flow  Testing  (DFT) 

Contract 

Hours 

Dollars 

Ground  System  Development 
Contractor 

702 

$70,200.00 

Operations  Contractor 

252 

$18,900.00 

Test  Asset 

0 

$0.00 

Satellite  Development  Contractor 

454 

$34,050.00 

$123,150.00 

Rehearsals 

Contract 

Hours 

Dollars 

Ground  System  Development 
Contractor 

120 

$12,000.00 

Operations  Contractor 

524 

$39,300.00 

Test  Asset 

0 

$0.00 

Satellite  Development  Contractor 

1200 

$90,000.00 

$141,300.00 

Week  In  The 

Life  Test  (WITF) 

Contract 

Hours 

Dollars 

Ground  System  Development 
Contractor 

40 

$4,000.00 

Operations  Contractor 

360 

$27,000.00 

Test  Asset 

0 

$0.00 

Satellite  Development  Contractor 

0 

$0.00 

$31,000.00 

Command  Validation  (CV) 

Contract 

Hours 

Dollars 

Ground  System  Development 
Contractor 

40 

$4,000.00 

Operations  Contractor 

106 

$7,950.00 
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Test  Asset 

0 

$0.00 

Satellite  Development  Contractor 

106 

$7,950.00 

$19,900.00 

Telemetry  Validation  (TV) 

Contract 

Hours 

Dollars 

Ground  System  Development 
Contractor 

40 

$4,000.00 

Operations  Contractor 

106 

$7,950.00 

Test  Asset 

0 

$0.00 

Satellite  Development  Contractor 

106 

$7,950.00 

$19,900.00 

Mission  Dress  Rehearsal  (MDR) 

Contract 

Hours 

Dollars 

Ground  System  Development 
Contractor 

24 

$7,200.00 

Operations  Contractor 

63.6 

$23,580.00 

Test  Asset 

0 

$0.00 

Satellite  Development  Contractor 

720 

$54,000.00 

$84,780.00 

Factory  Compatibility  Test  (FCT)  wit] 

iTSTR 

Contract 

Hours 

Dollars 

Ground  System  Development 
Contractor 

296 

$29,600.00 

Operations  Contractor 

412 

$30,900.00 

Test  Asset  Flight  Site  Survey 

$60,000.00 

Test  Asset  Flight  Operations 

$220,000.00 

Satellite  Development  Contractor 

708 

$53,100.00 

$393,600.00 

FCT  with  STGS-T 

Contract 

Hours 

Dollars 

Ground  System  Development 
Contractor 

296 

$29,600.00 

Operations  Contractor 

412 

$30,900.00 

Test  Asset  Flight  Site  Survey 

$60,000.00 

Test  Asset  Flight  Operations 

$160,000.00 

Satellite  Development  Contractor 

708 

$53,100.00 

$333,600.00 
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Launch  Based  Compatibility  Test  (LBCT)  at  ARTS  Site 

Contract 

Hours 

Dollars 

Ground  System  Development 
Contractor 

280 

$28,000.00 

Operations  Contractor 

412 

$30,900.00 

Test  Asset 

0 

$0.00 

Satellite  Development  Contractor 

1000 

$75,000.00 

$133,900.00 

LBCT  with  TSTR 

Contract 

Hours 

Dollars 

Ground  System  Development 
Contractor 

296 

$29,600.00 

Operations  Contractor 

412 

$30,900.00 

Test  Asset 

$220,000.00 

Satellite  Development  Contractor 

708 

$53,100.00 

$333,600.00 

LBCT  at  Launc 

[i  Site  with  STGS-T 

Contract 

Hours 

Dollars 

Ground  System  Development 
Contractor 

280 

$28,000.00 

Operations  Contractor 

412 

$30,900.00 

Test  Asset  Operations 

0 

$160,000.00 

Test  Asset  Flight  Site  Survey 

$60,000.00 

Satellite  Development  Contractor 

1000 

$75,000.00 

$353,900.00 

LBCT  at  Factory  with  STGS-T 

Contract 

Hours 

Dollars 

Ground  System  Development 
Contractor 

280 

$28,000.00 

Operations  Contractor 

412 

$30,900.00 

Test  Asset 

0 

$160,000.00 

Satellite  Development  Contractor 

708 

$53,100.00 

$272,000.00 
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EBCT  at  Eaunch  Site  with  TST] 

R 

Contract 

Hours 

Dollars 

Ground  System  Development 
Contractor 

296 

$29,600.00 

Operations  Contractor 

412 

$30,900.00 

Test  Asset  Operations 

$220,000.00 

Test  Asset  Elight  Site  Survey 

$60,000.00 

Satellite  Development  Contractor 

1000 

$75,000.00 

$415,500.00 

Along  with  costs,  we  asked  our  SMEs  to  assess  the  impacts  of  not  executing  each  of  the 
steps  in  the  validation  process.  We  consolidated  these  responses,  they  are  below: 

OAT-  Ground  system  would  not  be  validated  and  deemed  acceptable  to  conduct 
operations.  There  would  be  no  assurance  that  the  ground  system  meets  all  operational 
requirements. 

Exercises  -  Exercises  provide  a  means  to  indentify  project  deficiencies  related  to  the 
ground  system  or  mission  planning  processes;  without  these  events  potential  impacts  to 
the  project  schedule  and  cost  exist  due  to  the  discovery  of  the  deficiencies  later  in  the 
project  schedule. 

Data  Elow  Testing  -  Issues  with  ground  system  to  satellite  compatibility  would  not  be 
identified  at  earliest  opportunity.  The  later  in  the  validation  process,  compatibility  issues 
are  identified,  the  more  expensive  it  is  to  address  them. 

Rehearsals  -  Inadequate  preparedness  of  operations  support  staff  to  perform  mission 
operations;  unfamiliarity  with  the  ground  system  being  used  to  conduct  operations. 
Operational  impacts  to  functionality  of  ground  system  would  not  be  assessed. 

ECT  -  Inability  to  verify  correct  ARTS  IRON  database  configuration;  could  potentially 
result  in  loss  of  mission. 

WIPE  test  -  Conducted  to  identify  any  shortcomings  with  data  processing  over  an 
extended  period  of  time  and  to  assess  the  ground  system  stability  over  an  extended  period 
of  time.  Eor  some  missions  this  event  has  not  been  conducted  without  impact  to  the 
project. 

Command  validation  -  Significant  risk  of  inability  to  properly  command  the  satellite; 
could  result  in  loss  of  mission  or  data. 

Telemetry  validation  -  Inability  to  adequately  assess  the  health  and  safety  of  the  satellite; 
could  result  in  degraded  performance  or  loss  of  mission. 
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LBCT  -  satellite  eould  have  been  damaged  during  transport  to  the  launeh  faeility;  eould 
result  in  loss  of  mission. 

MDR  -  Validation  that  mission  operations  team  is  prepared  to  support  the  satellite  once 
on-orbit;  failure  to  conduct  this  event  could  result  in  launching  with  a  support  staff  that  is 
unprepared  for  launch. 

All  of  the  SMEs  pretty  much  agreed  on  the  impacts  of  not  executing  each  of  these  steps. 
We  have  included  these  in  the  second  survey  to  receive  final  concurrence.  From  the 
above  responses,  we  have  concluded  that  each  of  the  steps  in  the  validation  process  is 
very  important. 

STEP  2:  Identify  the  Risks  Associated  With  the  Compatibility  between  R&D  Satellite 
and  their  Ground  Systems,  Assign  Risks  to  Isolated  Steps  in  the  Validation  Process  and 
Define  Costs  Associated  with  Impacts  of  Risks: 

The  compatibility  between  a  satellite  and  its  ground  system  is  very  important.  Every 
satellite  program  office  assesses  and  monitors  the  risks  associated  with  this  compatibility. 

Each  step  in  the  validation  process  is  performed  in  order  to  mitigate  one  of  more  risks 
associated  with  the  compatibility  between  a  satellite  and  its  ground  system.  In  this  step, 
our  SMEs  identified  these  risks.  They  also  identified  what  steps  in  the  validation  process 
would  be  used  to  mitigate  these  risks.  Below  are  the  risks  identified  by  the  SMEs  and  the 
steps  in  the  validation  process  to  which  the  steps  map.  The  SMEs  also  identified  the 
probability  of  the  risk  occurring  before  performing  the  associated  steps  in  the  validation 
process,  the  probability  after  performing  the  steps  and  the  impact  of  the  risk  being 
realized.  The  following  table  was  used  to  assess  the  probabilities  and  impacts: 


> 

!5 

ns 

£ 

o 

o 

o 

£ 

0) 

£ 


Negligible 

Minor 

Moderate 

Serious 

91-100% 

61  -  90% 

41  -  60% 

11  -  40% 

0-10% 

Critical 


Consequences/Impact 
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Below  are  the  risks  the  SMEs  identified: 


Risk  1:  No  RF  Compatibility  between  the  Satellite  and  Ground  System 
Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  FCT  and  FBCT 
Risk  before  validation  step:  41-60% 

Risk  after  validation  step:  0-10% 

Severity:  Critical  -  Ground  System  is  unable  to  command  satellite,  possible  loss  of  range, 
range  rate  or  telemetry  data 

Cost  associated  with  risk  being  realized:  Possible  loss  of  msn  -  $100M 

Risk  2:  Configuration  incompatibility  between  RTS  &  SC  (i.e.,  ARTS  configuration, 
IRON  Database) 

Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  FCT  and  FBCT 
Risk  before  validation  step:  41-60% 

Risk  after  validation  step:  0-10% 

Severity:  Critical  -  unable  to  cmd,  possible  loss  of  range,  range  rate  or  telemetry  data 
Cost  associated  with  risk:  Possible  loss  of  msn  -  $100M 

Risk  3:  Command  database  incompatibility  and  errors  between  SC  &GS  (i.e.,  cmd 
database  problems) 

Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  Command  validation 
Risk  before  validation  step:  41-60% 

Risk  after  validation  step:  0  -10% 

Severity:  Serious  -  some  commands  may  not  work  properly  or  at  all 
Cost  associated  with  risk:  $1M 

Risk  4:  Telemetry  database  incompatibility  and  errors  between  SC  and  GS  (i.e.,  GS 
telemetry  database  problems) 

Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  Telemetry  Validation 
Risk  before  validation  step:  41-60% 

Risk  after  validation  step:  0-10% 

Severity:  Serious  -  some  telemetry  may  be  reported  incorrectly,  limits  may  be  set 
incorrectly 

Cost  associated  with  risk:  $1M 

Risk  5:  Ground  system  software  does  not  process  and  display  satellite  telemetry  correctly 
Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  Telemetry  Validation,  FCT,  DFT 
Risk  before  validation  step:  41-60% 

Risk  after  validation  step:  0-10% 

Severity:  Serious  -  Would  require  additional  software  drop,  could  cause  operator  to 
incorrectly  command  satellite  due  to  false  telemetry  processing 
Cost  associated  with  risk:  $500K 
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Risk  6:  Ground  system  software  does  not  construct  and  release  satellite  command 
correctly 

Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  Command  Validation,  FCT,  DFT 
Risk  before  validation  step:  1 1-20% 

Risk  after  validation  step:  0-10% 

Severity: 

Cost  associated  with  risk: 

Risk  7:  Ground  system  is  unable  correctly  post-pass  process  payload/mission  data 
correctly 

Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  WITL,  FCT 
Risk  before  validation  step:  41-60% 

Risk  after  validation  step:  0-10% 

Severity:  Moderate 
Cost  associated  with  risk: 

Risk  8:  Operational  or  data  latency  impacts  based  on  relationship  between  ground  system 
and  satellite  flight  software  (may  add  more  complexity  requiring  more  time  or  more 
resources  based  on  flight  software  handling  of  data) 

Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  Exercises  and  Rehearsals 
Risk  before  validation  step:  41-60% 

Risk  after  validation  step:  11-20% 

Severity:  Serious 

Cost  associated  with  risk:  $100K 

Risk  9:  A  satellite  manufacture  trying  something  new  with  command  format  which 
causes  compatibility  problems  between  ground  system  and  satellite. 

Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  DFT,  FCT,  LBCT,  Command 
Validation,  Telemetry  Validation 
Risk  before  validation  step:  61-80% 

Risk  after  validation  step:  11-20% 

Severity:  Serious  -  satellite  will  make  numerous  changes,  adding  cost  and  schedule 
(MUS  development) 

Cost  associated  with  risk:  $60-$200K  (MUS  dev  &  test  -  Dev  $50-$150K,  Test  $10- 
$50K) 

Risk  10:  Documentation  maturity  on  satellite  -  could  have  a  great  satellite,  but 
documentation  could  be  lacking 

Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  DFT 
Risk  before  validation  step:  61-80% 

Risk  after  validation  step:  11-20% 

Severity:  Minor  -  GS  will  be  built  poorly,  and  then  will  require  command  processing  and 

rework,  adding  cost  and  schedule 

Cost  associated  with  risk:  $24K  -  dev  $20K,  test  $4K 
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Risk  11:  Documentation  maturity  on  GS 

Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  DFT 

Risk  before  validation  step:  1 1-20% 

Risk  after  validation  step:  0-10% 

Severity:  Minor  -  satellite  manufacture  will  build  capability  that  ground  system  can’t 
handle,  ground  system  will  need  to  be  fixed.  Adds  cost  and  schedule 
Cost  associated  with  risk:  $24K  -  dev  $20K,  test  $4K 

Risk  12:  Satellite  is  not  mature  enough  in  development  to  have  important  compatibility 
parameters  defined. 

Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  Command  Validation  and 

Telemetry  Validation 

Risk  before  validation  step:  1 1-20% 

Risk  after  validation  step:  0-10% 

Severity:  Serious  -  less  mature  satellite  is  more  likely  to  have  changes  resulting  in 
changes  to  the  ground  system 

Cost  associated  with  risk:  $60-$200K  (MUS  dev  &  test  -  Dev  $50-$150K,  Test  $10- 
$50K) 

Risk  13:  Ground  System  is  not  mature  enough  in  development  to  have  important 
compatibility  parameters  defined. 

Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  FCT,  LBCT,  DFT 
Risk  before  validation  step:  1 1-20% 

Risk  after  validation  step:  0-10% 

Severity:  Moderate  -  Ground  System  may  not  meet  Satellite  schedule 
Cost  associated  with  risk:  $25K-$100K  (dev  -  $20K-$80K,  test  $5-$20K 

Risk  14:  Satellite  manufacturer  does  not  provide  telemetry  truth  data  for  ground  system 
DT&E  testing 

Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  DFT 
Risk  before  validation  step:  41-60% 

Risk  after  validation  step:  11-20% 

Severity:  Minor  -  telemetry  not  processed  correctly  -  rework  required  after  FCT  adding 
schedule  and  cost 

Cost  associated  with  risk:  $6K  (fix  database  -  $5K  -i-  $1K  test) 

Risk  15:  Satellite  does  not  provide  command  samples  for  ground  system  DT&E  testing 
Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  DPT 
Risk  before  validation  step:  41-60% 

Risk  after  validation  step:  11-20% 

Severity:  Minor  -  commands  not  processed  correctly  -  rework  required  after  PCT  adding 
schedule  and  cost 

Cost  associated  with  risk:  $6K  (fix  -  $5K  -i-  $1K  test) 
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Risk  16:  System  fails  to  support  all  operational  requirements  of  the  satellite 
Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  Exercises,  Rehearsals,  WETL 
Risk  before  validation  step:  1 1-20% 

Risk  after  validation  step:  0-10% 

Severity:  Moderate  -late  fix  -  schedule  slip 

Cost  associated  with  risk:  Anywhere  between  $200K  and  $2M  -depending  on  where  in 
the  readiness  process  the  problem  is  discovered 

Risk  17:  Ground  System  will  lose/corrupt  satellite  data 

Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  Command  &  Telemetry 
Validation 

Risk  before  validation  step:  41-60% 

Risk  after  validation  step:  0-10% 

Severity:  Minor  -  satellite  commanding  and  telemetry  will  be  erratic 
Cost  associated  with  risk:  $100K 

Risk  18:  Customer  delivered  planning  products  will  be  improperly  formatted  (i.e.  tasking 
files) 

Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  Telemetry  Validation 
Risk  before  validation  step:  0-10% 

Risk  after  validation  step:  0-10% 

Severity:  Minor  -  results  in  increased  ops  costs,  re -plan  contacts,  retransmit  commands, 
increased  maintenance  costs,  etc. 

Cost  associated  with  risk:  $100K  either  for  re-planning  activities  on  a  daily  basis,  or  a 
software  solution 

All  of  this  information  was  included  in  the  second  survey.  We  hope  to  gain  statistical 
confidence  that  this  is  a  correct  and  the  complete  list  of  risks.  We  hope  to  do  this  by 
gaining  the  majority  of  our  SMEs  concurrence.  The  second  survey  is  a  very  different 
format  than  the  first  survey.  Rather  than  asking  our  SMEs  to  give  input,  we  provide  the 
input  received  in  Survey  #1  and  ask  them  to  agree  or  disagree  and  then  provide 
comments  based  in  that.  All  subsequent  surveys  will  be  formatted  like  this. 
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Appendix  G:  Results  of  Survey  #2 


We  received  results  for  Survey  #2  from  the  same  eight  people:  three  military  members, 
one  government  civilian,  one  operations  contractor,  two  ground  system  contractors,  and 
one  independent  technical  advisor. 

Stepla:  The  Validation  Process:  This  first  subsection  of  each  survey  dealt  with  the 
validation  process  identified  on  page  in  chapter  three.  This  section  consisted  of  all  of  the 
same  questions  from  Survey  #1,  and  the  comments  provided  by  our  SMEs.  We  asked 
each  SME  to  either  agree  or  disagree  with  the  comments  provided  by  other  SMEs. 

Below  are  these  questions  and  the  responses. 

Question  1:  Are  these  the  right  steps  in  the  process? 

Responses  for  Survey  #1  with  Results  from  Survey  #2: 

1 .  Only  one  Week  in  the  Eife  Test  is  needed  -  All  SMEs  agreed  that  only  one  WITE  is 
required. 

2.  Rehearsal  and  training  products  are  part  of  the  training  for  the  operational  team  and 
are  not  part  of  the  overall  ground  system  validation  -  All  SMEs  with  Operational 
experience  agreed  that  Rehearsal  and  Exercises  are  an  important  part  of  the  validation 
process.  One  SME  disagreed. 

3.  Additional  testing  is  required  after  each  software  drop  and  needs  to  be  incorporated 
into  the  overall  process.  -  All  SMEs  agree  that  additional  testing  is  needed  after  each 
software  release. 

Question  2:  Are  the  steps  in  the  right  order? 

Responses  for  Survey  #1  with  Results  from  Survey  #2: 

1 .  Telemetry  validation  needs  to  occur  before  command  validation-  Only  one  SME  felt 
that  Telemetry  Validation  needs  to  be  completed  first,  the  rest  of  the  SMEs  feel  that 
CV  and  TV  are  usually  completed  together  or  it  doesn’t  matter. 

2.  Specific  to  missions,  frequently  operational  testing  occurs  much  later  than 
recommended  -  All  SMEs  but  one  believe  that  this  is  true;  the  one  SME  that 
disagreed  commented:  that  he  does  not  participate  in  OAT  and  therefore  did  not  have 
an  opinion. 

3.  Command  and  telemetry  validation  should  occur  after  each  software  drop  -  All  SMEs 
disagreed  with  this  statement.  Comments  we  got  included  that  testing  needs  to  be 
done,  but  understand  that  it  cannot  be  as  extensive  and  full  command  and  telemetry 
validation. 

4.  Operational  acceptance  testing  is  not  a  separate  function,  rather  part  of  the  entire 
process  -  We  did  not  get  a  consensus  from  our  SMEs  on  this  topic,  we  will  evaluate 
further  in  Survey  #3. 
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5.  Telemetry  and  command  validation  should  be  done  in  conjunction  with  either  data 
flow  tests  and/or  with  FCT  and  or  LBCT  -  Only  one  SME  disagreed  with  this 
statement.  He  is  the  independent  technical  advisor. 

6.  MDR  should  not  be  part  of  the  validation  effort  -  All  SMEs  agreed  with  this 
assessment 

7.  Prefers  order  -  DT&E,  ECT,  OAT,  Ex  1,  DET,  C&T  Validation,  Rehearsal  1,  WITE 
-  We  did  not  reach  consensus  on  this  order,  however  many  SMEs  commented  that  the 
order  of  the  validation  process  will  change  depending  on  the  mission.  This  is  the 
comment  we  will  be  including  Survey  #3. 

8.  Rehearsals/Exercises  have  a  tertiary  mission  of  validation,  but  primarily  used  for 
operational  training  and  thus  do  not  need  to  come  at  any  specific  time  in  the  process, 
but  must  be  completed,  and  in  fact  due  help  validate  the  system  (especially  if  a 
simulator  is  used  for  commanding)  -  All  SMEs  but  one  agreed  that  Rehearsal  and 
Exercises  are  an  important  part  of  the  validation  process,  one  SME  disagreed. 


Question  3  and  4:  Is  the  process  complete?  If  you  answered  no  above,  what  steps  in  the 

process  are  missing? 

Responses  for  Survey  #I  with  Results  from  Survey  #2: 

1.  Add  Day  in  the  Eife  Test  (DITE)  -  We  did  not  get  a  consensus  from  our  SMEs  on  this 
topic,  we  will  evaluate  further  in  Survey  #3. 

2.  Call  Operational  Testing:  System  Testing,  which  allows  you  to  leave  AESPC  out  of 
the  testing  loop  -  All  SMEs  agreed  with  this  assessment.  However,  we  will  not  be 
changing  the  name.  If  programs  wish  to  call  this  activity  system  testing  that  is  fine, 
but  the  objectives  are  to  ensure  that  the  system  meets  all  operational  requirements. 

3.  DT&E  is  missing  -  Only  one  SME  agreed  with  this  statement.  We  feel  that  DT&E  is 
a  part  of  verification  and  not  the  validation  process.  We  understand  that  DT&E  is 
important  but  feel  that  it  is  not  in  the  scope  of  this  thesis.  We  will  be  posing  this 
comment  in  Survey  #3 

Question  5:  Are  there  any  steps  you  don’t’  feel  are  part  of  the  validation  effort? 

Responses  for  Survey  #1  with  Results  from  Survey  #2: 

1.  Mission  Dress  Rehearsal  shouldn’t  be  a  part  of  the  validation  process  because  the 
ground  system  has  already  been  validated  by  this  point  -  All  SMEs  but  one  agreed 
that  MDR  is  not  part  of  the  Validation  Process,  one  SME  disagreed. 

2.  Only  steps  that  interface  the  ground  system  to  the  satellite  are  required  -  All  SMEs 
disagree  with  this  statement 

3.  EBCT  should  not  be  part  of  the  validation  effort  -  Only  one  SME  feels  this  is 
true. . .  .all  other  SMEs  feel  that  EBCT  is  an  important  step  in  the  validation  process. 
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4.  Rehearsals  2,  3,  etc  and  exercises  1,  2,  etc.  should  not  be  a  part  of  the  validation  effort 
-  Only  one  SME  agrees  with  this  assessment;  all  others  disagree.  From  this  SMEs 
comments,  we  feel  that  all  Rehearsals  and  Exercises  are  an  important  part  of 
validation,  but  we  may  combine  them  into  one  step  in  the  validation  process  since 
they  all  serve  the  same  function. 

5.  Eaunch  should  not  be  a  part  of  the  validation  effort.  -  All  SMEs  but  one  feel  that 
launch  is  not  part  of  validation.  We  think  that  our  SMEs  misunderstood  the  reason 
for  ending  the  process  with  launch.  We  did  not  intend  to  insinuate  that  launch  is  part 
of  the  validation  process  but  simply  that  the  validation  process  leads  to  launch.  The 
one  SME  that  disagreed  with  this  statement  did  bring  up  a  good  point  that  some 
requirements  cannot  be  tested  until  the  satellite  is  on-orbit. 

Step  lb:  Assign  appropriate  cost  to  each  activity:  This  second  subsection  deals  with 
assigning  a  cost  to  each  of  the  steps  in  the  validation  process.  We  included  the  cost 
estimate  that  calculated  based  on  past  and  current  program  actuals  in  the  second  survey. 
Below  are  the  cost  estimates  we  derived  and  the  SME  comments: 

1 .  OAT-  All  SMEs  agreed  with  our  cost  estimate,  assumptions  and  limitations  for  OAT 

2.  Exercises  -  All  SMEs  agreed  with  our  cost  estimate,  assumptions  and  limitations  for 
Exercises 

3.  DFT-  All  SMEs  agreed  with  our  cost  estimate,  assumptions  and  limitations  for  DFT 

4.  Rehearsals  -  All  SMEs  agreed  with  our  cost  estimate,  assumptions  and  limitations  for 
rehearsals 

5.  FCT  -  All  SMEs  agreed  with  our  cost  estimate,  assumptions  and  limitations  for  FCT 
with  the  exception  of  one  SME.  He  felt  that  this  cost  estimate  was  too  high.  We  feel 
that  although  he  is  correct  in  theory  many  missions  use  the  FCT  as  an  opportunity  to 
fully  test  the  satellite  and  ground  system  compatibility  end  to  end  that  therefore  the 
test  is  more  robust  and  more  expensive. 

6.  WITE  -  All  SMEs  agreed  with  our  cost  estimate,  assumptions  and  limitations  for 
WITE 

7.  CV  -  All  SMEs  agreed  with  our  cost  estimate,  assumptions  and  limitations  for  CV 

8.  TV  -  All  SMEs  agreed  with  our  cost  estimate,  assumptions  and  limitations  for  TV 

9.  EBCT  -  All  SMEs  agreed  with  our  cost  estimate,  assumptions  and  limitations  for 
EBCT 

10.  MDR-  All  SMEs  agreed  with  our  cost  estimate,  assumptions  and  limitations  for  MDR 

Along  with  costs,  we  asked  our  SMEs  to  assess  the  impacts  of  not  executing  each  of  the 
steps  in  the  validation  process.  We  consolidated  these  responses  from  Survey  #1  and 
asked  each  SME  to  agree  or  disagree  in  Survey  #2.  The  responses  are  below: 

OAT-  Ground  system  would  not  be  validated  and  deemed  acceptable  to  conduct 
operations.  There  would  be  no  assurance  that  the  ground  system  meets  all  operational 
requirements.  -  All  SMEs  agreed  with  the  impacts  of  not  executing  OAT. 
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Exercises  -  Exercises  provide  a  means  to  indentify  project  deficiencies  related  to  the 
ground  system  or  mission  planning  processes;  without  these  events  potential  impacts  to 
the  project  schedule  and  cost  exist  due  to  the  discovery  of  the  deficiencies  later  in  the 
project  schedule.  -  All  SMEs  agreed  with  the  impacts  of  not  executing  exercises. 

Data  Elow  Testing  -  Issues  with  ground  system  to  satellite  compatibility  would  not  be 
identified  at  earliest  opportunity.  The  later  in  the  validation  process,  compatibility  issues 
are  identified,  the  more  expensive  it  is  to  address  them.  -  All  SMEs  agreed  with  the 
impacts  of  not  executing  DET. 

Rehearsals  -  Inadequate  preparedness  of  operations  support  staff  to  perform  mission 
operations;  unfamiliarity  with  the  ground  system  being  used  to  conduct  operations. 
Operational  impacts  to  functionality  of  ground  system  would  not  be  assessed.  -  All  SMEs 
agreed  with  the  impacts  of  not  executing  rehearsals 

ECT  -  Inability  to  verify  correct  ARTS  IRON  database  configuration;  could  potentially 
result  in  loss  of  mission.  All  SMEs  agreed  with  the  impacts  of  not  executing  ECT  except 
one.  He  did  not  provide  any  comments  as  to  why. 

WITE  test  -  Conducted  to  identify  any  shortcomings  with  data  processing  over  an 
extended  period  of  time  and  to  assess  the  ground  system  stability  over  an  extended  period 
of  time.  Eor  some  missions  this  event  has  not  been  conducted  without  impact  to  the 
project.  -  All  SMEs  agreed  with  the  impacts  of  not  executing  WITE. 

Command  validation  -  Significant  risk  of  inability  to  properly  command  the  satellite; 
could  result  in  loss  of  mission  or  data.  -  All  SMEs  agreed  with  the  impacts  of  not 
executing  CV. 

Telemetry  validation  -  Inability  to  adequately  assess  the  health  and  safety  of  the  satellite; 
could  result  in  degraded  performance  or  loss  of  mission.  -  All  SMEs  agreed  with  the 
impacts  of  not  executing  TV. 

EBCT  -  satellite  could  have  been  damaged  during  transport  to  the  launch  facility;  could 
result  in  loss  of  mission.  -  All  SMEs  agreed  with  the  impacts  of  not  executing  EBCT. 

MDR  -  Validation  that  mission  operations  team  is  prepared  to  support  the  satellite  once 
on-orbit;  failure  to  conduct  this  event  could  result  in  launching  with  a  support  staff  that  is 
unprepared  for  launch.-  All  SMEs  agreed  with  the  impacts  of  not  executing  MDR. 

STEP  2:  Identify  the  Risks  Associated  with  the  Compatibility  between  R&D  Satellite  and 
their  Ground  Systems,  Assign  Risks  to  Isolated  Steps  in  the  Validation  Process  and 
Define  Costs  Associated  with  Impacts  of  Risks:  The  compatibility  between  a  satellite  and 
its  ground  system  is  very  important.  Every  satellite  program  office  assesses  and  monitors 
the  risks  associated  with  this  compatibility.  Each  step  in  the  validation  process  is 
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performed  in  order  to  mitigate  one  of  more  risks  associated  with  the  compatibility 
between  a  satellite  and  its  ground  system.  In  this  step,  our  SMEs  identified  these  risks. 
They  also  identified  what  steps  in  the  validation  process  would  be  used  to  mitigate  these 
risks.  Below  are  the  risks  identified  by  the  SMEs  and  the  steps  in  the  validation  process 
to  which  the  steps  map.  The  SMEs  also  identified  the  probability  of  the  risk  occurring 
before  performing  the  associated  steps  in  the  validation  process,  the  probability  after 
performing  the  steps  and  the  impact  of  the  risk  being  realized.  All  of  this  information 
was  gathered  in  Survey  #1,  in  Survey  #2  we  asked  the  SMEs  to  either  agree  or  disagree 
with  the  data.  The  following  table  was  used  to  assess  the  probabilities  and  impacts: 
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Below  are  the  risks  the  SMEs  identified: 


Risk  1:  No  RE  Compatibility  between  the  Satellite  and  Ground  System  -  All  SMEs 
agreed 

Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  ECT  and  EBCT-  All  SMEs 
agreed  with  exception  of  one;  no  comments  provided. 

Risk  before  validation  step:  41-60%-  All  SMEs  agreed  with  exception  of  one  no 
comments  provided. 

Risk  after  validation  step:  0-10%-  All  SMEs  agreed 

Severity:  Critical  -  Ground  System  is  unable  to  command  satellite,  possible  loss  of  range, 
range  rate  or  telemetry  data-  All  SMEs  agreed 

Cost  associated  with  risk  being  realized:  Possible  loss  of  msn  -  $100M  -  All  SMEs 
agreed 

Risk  2:  Configuration  incompatibility  between  RTS  &  SC  (i.e.,  ARTS  configuration, 
IRON  Database)  -  All  SMEs  agreed 

Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  ECT  and  EBCT-  All  SMEs 
agreed  with  the  exception  of  one. 
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Risk  before  validation  step:  41-60%  -  All  SMEs  agreed  with  exception  of  one. 

Risk  after  validation  step:  0-10%  -  All  SMEs  agreed 

Severity:  Critical  -  unable  to  cmd,  possible  loss  of  range,  range  rate  or  telemetry  data  - 
All  SMEs  agreed  with  exception  of  one. 

Cost  associated  with  risk:  Possible  loss  of  msn  -  $100M  -  All  SMEs  agreed  with 
exception  of  one. 

Risk  3:  Command  database  incompatibility  and  errors  between  SC  &GS  (i.e.,  cmd 
database  problems)  -  All  SMEs  agreed 

Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  Command  validation-  All  SMEs 
agreed 

Risk  before  validation  step:  41-60%  -  All  SMEs  agreed 

Risk  after  validation  step:  0  -10%-  All  SMEs  agreed  with  the  exception  of  one. 

Severity:  Serious  -  some  commands  may  not  work  properly  or  at  all-  All  SMEs  agreed 
Cost  associated  with  risk:  $1M  -  All  SMEs  agreed  with  exception  of  one. 

Risk  4:  Telemetry  database  incompatibility  and  errors  between  SC  and  GS  (i.e.,  GS 
telemetry  database  problems)  -  All  SMEs  agreed 

Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  Telemetry  Validation-  All  SMEs 
agreed 

Risk  before  validation  step:  41-60%  -  All  SMEs  agreed 
Risk  after  validation  step:  0-10%-  All  SMEs  agreed 

Severity:  Serious  -  some  telemetry  may  be  reported  incorrectly,  limits  may  be  set 
incorrectly-  All  SMEs  agreed 

Cost  associated  with  risk:  $1M-  All  SMEs  agreed  with  exception  of  one. 

Risk  5:  Ground  system  software  does  not  process  and  display  satellite  telemetry 
correctly-  All  SMEs  agreed 

Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  Telemetry  Validation,  ECT,  and 
DET-  All  SMEs  agreed 

Risk  before  validation  step:  41-60%-  All  SMEs  agreed  with  exception  of  one 
Risk  after  validation  step:  0-10%-  All  SMEs  agreed 

Severity:  Serious  -  Would  require  additional  software  drop,  could  cause  operator  to 
incorrectly  command  satellite  due  to  false  telemetry  processing  -  All  SMEs  agreed  with 
exception  of  one 

Cost  associated  with  risk:  $500K  -  All  SMEs  agreed  with  exception  of  one  who  thought 
that  the  impact  would  be  more  than  $1M 

Risk  6:  Ground  system  software  does  not  construct  and  release  satellite  command 
correctly-  All  SMEs  agreed 

Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  Command  Validation,  ECT,  and 
DET-  All  SMEs  agreed 

Risk  before  validation  step:  1 1-20%  -  All  SMEs  agreed  with  exception  of  one 
Risk  after  validation  step:  0-10%-  All  SMEs  agreed 
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Severity:  Critical  was  added  by  one  SME  no  one  else  commented;  we  hope  to  receive 
more  concurrence  in  Survey  #3 

Cost  associated  with  risk:  Loss  of  mission;  $100M  was  added  by  one  SME  no  one  else 
commented;  we  hope  to  receive  more  concurrence  in  Survey  #3 

Risk  7:  Ground  system  is  unable  correctly  post-pass  process  payload/mission  data 
correctly  -  All  SMEs  agreed 

Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  WITL,  ECT-  All  SMEs  agreed 
Risk  before  validation  step:  41-60%-  All  SMEs  agreed  with  exception  of  one 
Risk  after  validation  step:  0-10%-  All  SMEs  agreed 
Severity:  Moderate-  All  SMEs  agreed 

Cost  associated  with  risk:  delay  in  mission  data  <$200K  was  added  by  one  SME  no  one 
else  commented;  we  hope  to  receive  more  concurrence  in  Survey  #3 

Risk  8:  Operational  or  data  latency  impacts  based  on  relationship  between  ground  system 
and  satellite  flight  software  (may  add  more  complexity  requiring  more  time  or  more 
resources  based  on  flight  software  handling  of  data)  -  All  SMEs  agreed 
Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  Exercises  and  Rehearsals-  All 
SMEs  agreed 

Risk  before  validation  step:  41-60%  -  All  SMEs  agreed  with  exception  of  one 
Risk  after  validation  step:  11-20%  -  All  SMEs  agreed  with  exception  of  one 
Severity:  Serious-  All  SMEs  agreed  with  exception  of  one 

Cost  associated  with  risk:  $100K  was  added  by  one  SME  no  one  else  commented;  we 
hope  to  receive  more  concurrence  in  Survey  #3 

Risk  9:  A  satellite  manufacture  trying  something  new  with  command  format  which 
causes  compatibility  problems  between  ground  system  and  satellite.  -  All  SMEs  agreed 
Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  DPT,  ECT,  LBCT,  Command 
Validation,  and  Telemetry  Validation-  All  SMEs  agreed 
Risk  before  validation  step:  61-80%  -  All  SMEs  agreed 
Risk  after  validation  step:  11-20%  -  All  SMEs  agreed 

Severity:  Serious  -  satellite  will  make  numerous  changes,  adding  cost  and  schedule 
(MUS  development)  -  All  SMEs  agreed 

Cost  associated  with  risk:  $60-$200K  (MUS  dev  &  test  -  Dev  $50-$150K,  Test  $10- 
$50K)  -  All  SMEs  agreed 

Risk  10:  Documentation  maturity  on  satellite  -  could  have  a  great  satellite,  but 
documentation  could  be  lacking-  All  SMEs  agreed 

Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  DPT-  All  SMEs  agreed  with 
exception  of  one 

Risk  before  validation  step:  61-80%  -  All  SMEs  agreed 

Risk  after  validation  step:  11-20%-  All  SMEs  agreed  with  exception  of  one 

Severity:  Minor  -  GS  will  be  built  poorly,  and  then  will  require  command  processing  and 

rework,  adding  cost  and  schedule-  All  SMEs  agreed  with  exception  of  one 
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Cost  associated  with  risk:  $24K  -  dev  $20K,  test  $4K-  All  SMEs  agreed  with  exception 
of  one 

Risk  1 1 :  Documentation  maturity  on  GS-  All  SMEs  agreed 

Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  DET-  All  SMEs  agreed  with 
exception  of  one 

Risk  before  validation  step:  1 1-20%-  All  SMEs  agreed  with  exception  of  one 
Risk  after  validation  step:  0-10%-  All  SMEs  agreed  with  exception  of  one 
Severity:  Minor  -  satellite  manufacture  will  build  capability  that  ground  system  can’t 
handle,  ground  system  will  need  to  be  fixed.  Adds  cost  and  schedule-  All  SMEs  agreed 
Cost  associated  with  risk:  $24K  -  dev  $20K,  test  $4K-  All  SMEs  agreed  with  exception 
of  one  who  thinks  the  cost  would  be  more. 

Risk  12:  Satellite  is  not  mature  enough  in  development  to  have  important  compatibility 
parameters  defined.  -  All  SMEs  agreed 

Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  Command  Validation  and 
Telemetry  Validation-  All  SMEs  agreed  with  exception  of  one 
Risk  before  validation  step:  1 1-20%-  All  SMEs  agreed  with  exception  of  one 
Risk  after  validation  step:  0-10%-  All  SMEs  agreed  with  exception  of  one 
Severity:  Serious  -  less  mature  satellite  is  more  likely  to  have  changes  resulting  in 
changes  to  the  ground  system-  All  SMEs  agreed 

Cost  associated  with  risk:  $60-$200K  (MUS  dev  &  test  -  Dev  $50-$150K,  Test  $10- 
$50K)  -  All  SMEs  agreed 

Risk  13:  Ground  System  is  not  mature  enough  in  development  to  have  important 
compatibility  parameters  defined.  -  All  SMEs  agreed 

Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  ECT,  EBCT,  and  DET-  All  SMEs 
agreed 

Risk  before  validation  step:  1 1-20%-  All  SMEs  agreed  with  exception  of  one 

Risk  after  validation  step:  0-10%-  All  SMEs  agreed  with  exception  of  one 

Severity:  Moderate  -  Ground  System  may  not  meet  Satellite  schedule-  All  SMEs  agreed 

Cost  associated  with  risk:  $25K-$100K  (dev  -  $20K-$80K,  test  $5-$20K-  All  SMEs 

agreed 

Risk  14:  Satellite  manufacturer  does  not  provide  telemetry  truth  data  for  ground  system 
DT&E  testing-  All  SMEs  agreed 

Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  DPT-  All  SMEs  agreed  with 
exception  of  one 

Risk  before  validation  step:  41-60%-  All  SMEs  agreed 

Risk  after  validation  step:  11-20%-  All  SMEs  agreed  with  exception  of  one 

Severity:  Minor  -  telemetry  not  processed  correctly  -  rework  required  after  ECT  adding 

schedule  and  cost-  All  SMEs  agreed  with  exception  of  one. 

Cost  associated  with  risk:  $6K  (fix  database  -  $5K  -i-  $1K  test)  -  All  SMEs  agreed 
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Risk  15:  Satellite  does  not  provide  command  samples  for  ground  system  DT&E  testing  - 
All  SMEs  agreed  with  exception  of  one 

Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  DET-  All  SMEs  agreed  with 
exception  of  one 

Risk  before  validation  step:  41-60%-  All  SMEs  agreed  with  exception  of  one 

Risk  after  validation  step:  11-20%-  All  SMEs  agreed  with  exception  of  one 

Severity:  Minor  -  commands  not  processed  correctly  -  rework  required  after  ECT  adding 

schedule  and  cost-  All  SMEs  agreed  with  exception  of  one 

Cost  associated  with  risk:  $6K  (fix  -  $5K  -i-  $1K  test)  -  All  SMEs  agreed 

Risk  16:  System  fails  to  support  all  operational  requirements  of  the  satellite-  All  SMEs 
agreed 

Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  Exercises,  Rehearsals,  WITE-  All 
SMEs  disagreed 

Risk  before  validation  step:  1 1-20%-  All  SMEs  agreed  with  exception  of  one 
Risk  after  validation  step:  0-10%-  All  SMEs  agreed 
Severity:  Moderate  -late  fix  -  schedule  slip-  All  SMEs  agreed 

Cost  associated  with  risk:  Anywhere  between  $200K  and  $2M  -depending  on  where  in 
the  readiness  process  the  problem  is  discovered  -  All  SMEs  agreed 

Risk  17:  Ground  System  will  lose/corrupt  satellite  data-  All  SMEs  agreed 

Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  Command  &  Telemetry 

Validation-  All  SMEs  agreed  with  exception  of  one 

Risk  before  validation  step:  41-60%-  All  SMEs  agreed 

Risk  after  validation  step:  0-10%-  All  SMEs  agreed  with  exception  of  one 

Severity:  Minor  -  satellite  commanding  and  telemetry  will  be  erratic-  All  SMEs  agreed 

with  exception  of  one 

Cost  associated  with  risk:  $100K  was  added  by  one  SME  no  one  else  commented;  we 
hope  to  receive  more  concurrence  in  Survey  #3 

Risk  18:  Customer  delivered  planning  products  will  be  improperly  formatted  (i.e.  tasking 
files)  -  All  SMEs  agreed 

Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  Telemetry  Validation  -  All  SMEs 
disagreed 

Risk  before  validation  step:  0-10%-  All  SMEs  disagreed 
Risk  after  validation  step:  0-10%  -  All  SMEs  agreed 

Severity:  Minor  -  results  in  increased  ops  costs,  re -plan  contacts,  retransmit  commands, 
increased  maintenance  costs,  etc.  -  All  SMEs  agreed 

Cost  associated  with  risk:  $100K  either  for  re-planning  activities  on  a  daily  basis,  or  a 
software  solutions  lOOK  was  added  by  one  SME  no  one  else  commented;  we  hope  to 
receive  more  concurrence  in  Survey  #3 

We  got  very  few  comments  in  the  risk  section.  We  will  add  any  disagreements  to  Survey 
#3  and  hope  to  receive  more  concurrence  and  clarification  in  Survey  #3. 
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Appendix  H:  Results  of  Survey  #3 


We  received  results  for  Survey  #3  from  the  same  eight  people:  three  military  members, 
one  government  civilian,  one  operations  contractor,  two  ground  system  contractors,  and 
one  and  one  independent  technical  advisor. 

**  The  only  comments  we  included  in  Survey  #3  were  ones  that  did  not  have  unanimous 
concurrence  on  Survey  #2.** 

The  Validation  Process:  This  first  subsection  of  each  survey  dealt  with  the  validation 
process  identified  on  page  in  chapter  three.  This  section  consisted  of  all  of  the  same 
questions  from  Survey  #2  with  comments  that  did  not  receive  100%  concurrence  in 
Survey  #2. 

Question  1:  Are  these  the  right  steps  in  the  process? 

***No  comments  were  included  pertaining  to  this  question  in  Survey  #3  because  we 
believe  that  we  received  adequate  concurrence*** 

Question  2:  Are  the  steps  in  the  right  order? 

1 .  Operational  acceptance  testing  is  not  a  separate  function,  rather  part  of  the  entire 
process  -  We  still  did  not  receive  concurrence  on  this  subject  in  Survey  #3.  In  fact 
none  of  the  SMEs  changed  their  responses. 

Response: 

1 .  Only  one  SME  felt  strongly  that  OAT  should  not  be  accomplished  as  a  separate  test, 
therefore  we  will  include  it  in  the  process. 

2.  Prefers  order  -  DT&E,  EOT,  OAT,  Ex  1,  DET,  C&T  Validation,  Reh  1,  WITE  -  All 
of  our  SMEs  agreed  that  the  order  of  the  validation  process  will  vary  from  mission  to 
mission.  Therefore  we  are  going  to  continue  with  the  original  order  (with  the 
omissions  and  additions  from  our  SMEs)  and  discuss  this  in  our  discussion. 


Question  3  and  4:  Is  the  process  complete?  If  you  answered  no  above,  what  steps  in  the 
process  are  missing? 

1 .  Add  Day  in  the  Eife  Test  (DITE) 

Response:  All  SMEs  agreed  that  a  DITE  should  be  added  to  the  process. 

Question  5:  Are  there  any  steps  you  don’t’  feel  are  part  of  the  validation  effort? 
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***No  comments  were  included  pertaining  to  this  question  in  Survey  #3  because  we 
believe  that  we  received  adequate  concurrence***. 

Assign  appropriate  cost  to  each  activity:  This  second  subsection  deals  with  assigning  a 
cost  to  each  of  the  steps  in  the  validation  process.  We  included  the  cost  estimate  that 
calculated  based  on  past  and  current  program  actuals  in  the  second  survey.  Below  are  the 
cost  estimates  we  derived  and  the  SME  comments: 

1 .  WITL  -  All  SMEs  agreed  with  our  cost  estimate,  assumptions  and  limitations  for 
WITE 

Response:  All  SMEs  felt  that  the  cost  was  too  low  because  we  did  not  include  satellite 
manufacturer  hours  and  costs.  We  included  these  and  here  is  the  new  WITE  costs. 


Week  In  The  Eife  Test  (WIPE) 

Contract 

Hours 

Dollars 

Ground  System  Contract 

40 

$4,000.00 

Operations  Contract 

360 

$27,000.00 

Mobile  Range  Elight 

0 

$0.00 

Satellite  Developer 

360 

$27,000.00 

$58,000.00 

2.  CV  -  All  SMEs  agreed  with  our  cost  estimate,  assumptions  and  limitations  for  CV 

Response:  Only  SME  comment  was  that  this  should  take  3  people  one  week  which  is 
consist  with  our  original  estimate  so  we  will  keep  our  original  estimate. 

Along  with  costs,  we  asked  our  SMEs  to  assess  the  impacts  of  not  executing  each  of  the 
steps  in  the  validation  process.  We  only  included  the  responses  from  Survey  #2  in 
Survey  #3  that  did  not  have  unanimous  agreement. 

ECT  -  Inability  to  verify  correct  ARTS  IRON  database  configuration;  could  potentially 
result  in  loss  of  mission.  All  SMEs  agreed  with  the  impacts  of  not  executing  ECT 

EBCT  -  satellite  could  have  been  damaged  during  transport  to  the  launch  facility;  could 
result  in  loss  of  mission.  -  All  SMEs  agreed  with  the  impacts  of  not  executing  EBCT. 

STEP  2:  Identify  the  Risks  Associated  With  the  Compatibility  between  R&D  Satellite 
and  their  Ground  Systems,  Assign  Risks  to  Isolated  Steps  in  the  Validation  Process  and 
Define  Costs  Associated  with  Impacts  of  Risks:  The  compatibility  between  a  satellite  and 
its  ground  system  is  very  important.  Every  satellite  program  office  assesses  and  monitors 
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the  risks  associated  with  this  compatibility.  Each  step  in  the  validation  process  is 
performed  in  order  to  mitigate  one  of  more  risks  associated  with  the  compatibility 
between  a  satellite  and  its  ground  system.  In  this  step,  our  SMEs  identified  these  risks. 
They  also  identified  what  steps  in  the  validation  process  would  be  used  to  mitigate  these 
risks.  Below  are  the  risks  identified  by  the  SMEs  and  the  steps  in  the  validation  process 
to  which  the  steps  map.  The  SMEs  also  identified  the  probability  of  the  risk  occurring 
before  performing  the  associated  steps  in  the  validation  process,  the  probability  after 
performing  the  steps  and  the  impact  of  the  risk  being  realized.  Only  the  comments 
without  100%  agreement  in  Survey  #2  were  included  in  Survey  #3. 
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Consequences/Impact 


Below  are  the  risks  the  SMEs  identified: 

Risk  1:  No  RE  Compatibility  between  the  Satellite  and  Ground  System 

Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  ECT  and  EBCT-  All  SMEs 

agreed  that  this  was  a  good  list  of  steps 

Risk  before  validation  step:  41-60%-  All  SMEs  agreed  that  this  was  a  good  assessment 

Risk  2:  Configuration  incompatibility  between  RTS  &  SC  (i.e.,  ARTS  configuration, 
IRON  Database) 

Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  ECT  and  EBCT-  All  SMEs 
agreed  however,  we  should  include  in  our  discussion  that  EBCT  importance  drops  with  a 
successful  ECT. 

Risk  before  validation  step:  41-60%  -  All  SMEs  agreed  risks  probability  is  acceptable. 
Severity:  Critical  -  unable  to  command,  possible  loss  of  range,  range  rate  or  telemetry 
data  -  All  SMEs  agreed  severity  is  acceptable. 

Cost  associated  with  risk:  Possible  loss  of  msn  -  $100M  -  All  SMEs  agreed  the  costs  is 
acceptable. 
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Risk  3:  Command  database  incompatibility  and  errors  between  SC  &GS  (i.e.,  command 
database  problems)  -  All  SMEs  agreed 

Cost  associated  with  risk:  $1M  -  All  SMEs  agreed  with  costs  associated  with  this  risk 

Risk  4:  Telemetry  database  incompatibility  and  errors  between  SC  and  GS  (i.e.,  GS 
telemetry  database  problems) 

Cost  associated  with  risk:  $1M-  All  SMEs  but  one  agree  that  this  is  appropriate;  the  one 
SME  is  not  changing  his  opinion  so  we  will  go  with  the  majority. 

Risk  5:  Ground  system  software  does  not  process  and  display  satellite  telemetry  correctly 
Risk  before  validation  step:  41-60%  -  All  SMEs  agree  this  is  correct 
Severity:  Serious  -  Would  require  additional  software  drop,  could  cause  operator  to 
incorrectly  -  All  SMEs  agree  this  is  correct 

Risk  6:  Ground  system  software  does  not  construct  and  release  satellite  command 
correctly 

Risk  before  validation  step:  1 1-40%  -  All  SMEs  but  one  agree  that  this  is  appropriate;  the 
one  SME  is  not  changing  his  opinion  so  we  will  go  with  the  majority 

Risk  7:  Ground  system  is  unable  correctly  post-pass  process  payload/mission  data 
correctly 

Risk  before  validation  step:  41-60%-  All  SMEs  agreed  with  exception  of  one-  All  SMEs 
but  one  agree  that  this  is  appropriate;  the  one  SME  is  not  changing  his  opinion  so  we  will 
go  with  the  majority 

Risk  8:  Operational  or  data  latency  impacts  based  on  relationship  between  ground  system 
and  satellite  flight  software  (may  add  more  complexity  requiring  more  time  or  more 
resources  based  on  flight  software  handling  of  data)  -  All  SMEs  agreed 
Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  Exercises  and  Rehearsals-  SMEs 
agree  that  we  should  add  DET,  ECT  and  EBCT 

Risk  before  validation  step:  41-60%  -  All  SMEs  agreed  this  is  acceptable 
Risk  after  validation  step:  11-20%  -  All  SMEs  agreed  this  is  acceptable 
Severity:  Serious-  All  SMEs  agreed  this  is  acceptable 

Risk  9:  A  satellite  manufacture  trying  something  new  with  command  format  which 
causes  compatibility  problems  between  ground  system  and  satellite. 

Risk  after  validation  step:  11-20%  -  All  SMEs  agreed  this  is  acceptable 

Risk  10:  Documentation  maturity  on  satellite  -  could  have  a  great  satellite,  but 
documentation  could  be  lacking 

Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  DPT-  SMEs  feel  we  should  add 
ECT  and  EBCT 

Risk  after  validation  step:  0-10%-  All  SMEs  agreed  with  exception  of  one 
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Severity:  Minor  -  GS  will  be  built  poorly,  and  then  will  require  command  processing  and 
rework,  adding  cost  and  schedule-  All  SMEs  agreed  is  acceptable 

Risk  1 1 :  Documentation  maturity  on  GS 

Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  DFT-  All  SMEs  agreed  this  is 
acceptable 

Risk  before  validation  step:  1 1-40%-  All  SMEs  agreed  this  is  acceptable 
Risk  after  validation  step:  0-10%-  All  SMEs  agreed  this  is  acceptable 
***Bad  training,  incorrect  ops  procedures  and  disillusioned  operators  can  add  to  this 
problem.  *** 

Cost  associated  with  risk:  $24K  -  dev  $20K,  test  $4K-  All  SMEs  agreed  this  is 
acceptable 

Risk  12:  Satellite  is  not  mature  enough  in  development  to  have  important  compatibility 
parameters  defined. 

Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  Command  Validation  and 
Telemetry  Validation-  All  SMEs  agreed  this  is  acceptable 

Risk  before  validation  step:  1 1-20%-  All  SMEs  but  one  agree  that  this  is  appropriate;  the 
one  SME  is  not  changing  his  opinion  so  we  will  go  with  the  majority. 

Risk  after  validation  step:  0-10%-  All  SMEs  but  one  agree  that  this  is  appropriate;  the 
one  SME  is  not  changing  his  opinion  so  we  will  go  with  the  majority. 

Risk  13:  Ground  System  is  not  mature  enough  in  development  to  have  important 
compatibility  parameters  defined. 

Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  FCT,  EBCT,  and  DFT-  All  SMEs 
agreed 

Risk  before  validation  step:  1 1-20%-  All  SMEs  but  one  agree  that  this  is  appropriate;  the 
one  SME  is  not  changing  his  opinion  so  we  will  go  with  the  majority. 

Risk  after  validation  step:  0-10%  -  All  SMEs  but  one  agree  that  this  is  appropriate;  the 
one  SME  is  not  changing  his  opinion  so  we  will  go  with  the  majority. 

Risk  14:  Satellite  manufacturer  does  not  provide  telemetry  truth  data  for  ground  system 
DT&E  testing-  All  SMEs  agreed 

Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  DFT-  All  SMEs  agreed  to  add 
TV,  FCT,  and  EBCT 

Risk  after  validation  step:  11-20%-  All  SMEs  but  one  agree  that  this  is  appropriate;  the 
one  SME  is  not  changing  his  opinion  so  we  will  go  with  the  majority. 

Severity:  Minor  -  telemetry  not  processed  correctly  -  rework  required  after  FCT  adding 
schedule  and  cost-  All  SMEs  agreed  this  is  acceptable. 
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Risk  15:  Satellite  does  not  provide  command  samples  for  ground  system  DT&E  testing 
Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  DFT-  SMEs  feel  that  we  should 
addCV 

Risk  before  validation  step:  41-60%-  All  SMEs  agreed  this  is  acceptable 

Risk  after  validation  step:  11-20%-  All  SMEs  agreed  this  is  acceptable 

Severity:  Minor  -  commands  not  processed  correctly  -  rework  required  after  ECT  adding 

schedule  and  cost-  All  SMEs  but  one  agree  that  this  is  appropriate;  the  one  SME  is  not 

changing  his  opinion  so  we  will  go  with  the  majority. 

Risk  16:  System  fails  to  support  all  operational  requirements  of  the  satellite 

Risk  before  validation  step:  1 1-40%-  All  SMEs  but  one  agree  that  this  is  appropriate;  the 

one  SME  is  not  changing  his  opinion  so  we  will  go  with  the  majority. 

Risk  after  validation  step:  0-10%-  All  SMEs  agreed  this  is  acceptable 

Risk  17:  Ground  System  will  lose/corrupt  satellite  data 

Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  Command  &  Telemetry 
Validation-  All  SMEs  agreed  with  exception  of  one;  the  one  SME  is  not  changing  his 
opinion  so  we  will  go  with  the  majority. 

Risk  after  validation  step:  0-10%-  All  SMEs  agreed  with  exception  of  one  the  one  SME 
is  not  changing  his  opinion  so  we  will  go  with  the  majority 

Severity:  Minor  -  satellite  commanding  and  telemetry  will  be  erratic-  All  SMEs  agreed 
this  is  acceptable 

Risk  18:  Customer  delivered  planning  products  will  be  improperly  formatted  (i.e.  tasking 
files)  -  All  SMEs  agreed 

Steps  in  the  Validation  Process  that  Mitigate  this  Risk:  Telemetry  Validation  -  All  SMEs 

agreed  that  we  should  add  WIPE,  TV,  exercises  and  rehearsals 

Risk  before  validation  step:  0-10%-  All  SMEs  felt  this  was  acceptable 
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Appendix  I:  Results  from  Simulation  Runs 


Program  Office  Proposed  Validation  Strategy 


Option  1  Trial  1:  PM 


0.6 

0.5 

0.4 

0.3 

0.2 

0.1 

0 


“I — I — I — I — I — I — I — I — I — I — n 


■r 


■r 


^  ^  <o^  ^5  <o^ 

o??’  C^'^■  0>-  0?>-  eO-  ,  <0-  rO-  4«0-  o??'  6<o-  c^- 

0?>'  cP'  (^'  0>'  C^'  (o'O'  O?’'  C?>'  <>$0'  0?>'  (^'  (,$5'  P>'  <^'  (Jp'  0?>'  PP' 

^V  cO.'  cjX'  oV  qV  ^ 

^  V  V  V 


Option  1  Trial  2:  PM 
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Option  1  Trial  3:  PM 
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Option  1  All :  Author 
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IRRT  Proposed  Validation  Strategy 


Option  2  Trial  1:  IRRT 
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Option  2  Trial  2;  IRRT 
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Validation  Strategy  Costs 

Mean 

Range 

Standard  Deviation 

Median 

Option  2 Trial  1:  IRRT 

$1,517,150.00 

$17,492,726.15 

$201,000,000.00 

$37,479,079.62 

$2,017,150.00 

Option  2 Trial  2:  IRRT 

$1,517,150.00 

$16,981,410.52 

$202,071,000.00 

$38,403,474.14 

$1,773,150.00 

Option  2 Trial  3:  IRRT 

$1,517,150.00 

$16,981,410.52 

$202,071,000.00 

$38,403,474.14 

$1,773,150.00 

Option  2  All:  IRRT 

$1,517,150.00 

$17,130,988.89 

$202,071,000.00 

$38,064,722.27 

$1,817,150.00 
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Author  Proposed  Validation  Strategy 
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Option  3  Trial  3:  Author 
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Option  3  All:  Author 
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Validation  Strategy  Costs 

Mean 

Range 

Standard  Deviation 

Median 

Option  3Trial  1:  Author 

$1,031,050.00 

$18,784,992.00 

$201,321,000.00 

$39,572,532.20 

$1,405,050.00 

Option  BTrial  2:  Author 

$1,031,050.00 

$18,770,949.00 

$201,248,000.00 

$41,266,558.94 

$1,596,050.00 

Option  BTrial  B:  Author 

$1,031,050.00 

$16,982,973.00 

$201,165,000.00 

$39,584,450.95 

$1,331,050.00 

Option  BAM:  Author 

$1,031,050.00 

$18,179,638.00 

$201,321,000.00 

$40,144,600.32 

$1,405,050.00 
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Validation  Strategy  Comparison 
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- Option  1:  PM 

Cumuiative 


- Option  2:  iRRT 

Cumuiative 


Option  3: 

Author 

Cumuiative 


Validation  Strategy  Costs 

Mean 

Range 

Standard  Deviation 

Median 

Option  1  Aii:  PM 

$1,324,850.00 

$18,970,647.13 

$301,129,400.00 

$40,253,166.62 

$2,924,850.00 

Option  2  Aii:  IRRT 

$1,517,150.00 

$17,130,988.89 

$202,071,000.00 

$38,064,722.27 

$1,817,150.00 

Option  3  All;  Author 

$1,031,050.00 

$18,179,638.00 

$201,321,000.00 

$40,144,600.32 

$1,405,050.00 
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